Lead Azure Databricks Platform Engineer / Architect

Lead Azure Databricks Platform Engineer / Architect

Hands-on Platform Engineering | Serverless | FinOps | POSIT/RStudio Migration

6 Month contract

Inside IR35 - 500 a day

London/Hybrid

Role Purpose

We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. The role combines architecture with direct implementation: the successful candidate must be able to configure, develop, troubleshoot and optimise Azure Databricks rather than operate only at design or governance level.

The role is centred on three outcomes: enabling and optimising Databricks Serverless, strengthening FinOps and platform controls, and enhancing the Databricks Discovery Zone to support workloads currently delivered through POSIT/RStudio.

Key Responsibilities

1. Databricks Serverless Enablement and Optimisation

Assess existing workloads and determine suitability for Serverless, classic, job or interactive compute based on duration, utilisation, SLA, concurrency, performance and cost.

Enable and configure Serverless for appropriate jobs, SQL workloads, notebooks, analytical processing and data pipelines.

Establish workload-placement guidance, including when Serverless is not economical for predictable, heavy or continuously running workloads.

Implement compute policies, autoscaling, quotas, budget controls and operational guardrails.

Measure cost and performance outcomes, identify idle or oversized compute, and recommend optimisation actions.

2. FinOps and Enterprise Platform Controls

Define and embed a practical FinOps operating model covering ownership, accountability, projects, environments, teams, applications and cost centres.

Implement mandatory tagging and integrate validation into CI/CD so non-compliant resources are prevented from being provisioned.

Provide granular cost attribution by workspace, project, application, workload, job and team/user where technically appropriate.

Implement budget policies, thresholds, proactive alerts and usage reporting to prevent uncontrolled spend.

Use platform usage and billing data to identify idle compute, inefficient workloads, unnecessary storage/data movement and cost anomalies.

3. Databricks Discovery Zone and POSIT/RStudio Migration

Enhance the Databricks Discovery Zone to support migration from POSIT/RStudio

Enable application deployment, secure API integrations, external data ingestion, LLM integration, scheduling, BI connectivity, local IDE-based development and operational reporting.

Define reusable onboarding and migration patterns that reduce technology sprawl while improving security, supportability and delivery speed.

4. Data Engineering and Integration

Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake.

Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls.

Design reusable integration patterns for REST APIs, SaaS platforms, databases, files, object storage, document repositories, enterprise applications and public/external data providers.

Implement secure authentication and credential handling for external and internal integrations.

Build end-to-end data flows from source through governed ingestion and curated layers to BI, ML or application consumption.

Required Hands-on Technical Skills

Deep hands-on Azure Databricks implementation and troubleshooting

Databricks Serverless and compute/workload optimisation

Azure identity, networking, security, secrets, monitoring and private connectivity

Databricks SQL, Delta Lake and performance optimisation

Python, PySpark and SQL

Jobs/workflows, incremental processing, CDC and data quality

REST/API and external data integration patterns

FinOps, cost attribution, tagging, budgets, monitoring and operational support

Experience and Candidate Profile

Significant experience delivering enterprise Azure Databricks platforms in production environments.

Demonstrable ability to move between architecture, implementation, debugging and optimisation without depending entirely on specialist engineering teams.

Strong understanding of platform security, data governance, operational support and controlled delivery in regulated or complex enterprises.

Experience working collaboratively with data engineers, data scientists, architects, security teams, platform teams and business stakeholders.

Clear communication skills and the ability to document standards, patterns, decisions and operational guidance.

Highly Desirable but not Mandatory

POSIT/RStudio migration or consolidation experience.

Migration of analytical/data science workloads (convert and migrate R development/Libraries to Databricks).

AI/ML, LLM integration, model lifecycle, RAG/vector retrieval or model-serving experience.

Large-scale enterprise platform transformation and regulated-industry experience.

Strong cost optimisation and FinOps delivery experience across Azure and Databricks