Lead Azure Databricks Engineer

Lead Azure Databricks Engineer - £80-90k plus 15% cash flex which can be used to buy additional benefits or taken as cash each month plus bonus - London hybrid

Fantastic Opportunity to join a well know Global Consulting firm's Insights and Data Practice.

Our client is seeking an exceptional Lead Azure Databricks Engineer to architect, build, and optimise enterprise‑scale cloud data platforms. This hands‑on leadership role is central to shaping their Azure and Databricks strategy, enabling advanced analytics, regulatory compliance, and robust engineering across the organisation. The ideal candidate brings deep technical expertise, strong delivery ownership, and proven experience operating within the highly regulated Lloyd's of London market.

Hybrid working:

The places that you work from day to day will vary according to your role, your needs, and those of the business; it will be a blend of Company offices, client sites, and your home; noting that you will be unable to work at home 100% of the time.

Your Role:

Design, develop, and optimise large‑scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog.
Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance, and Solvency II teams to meet Lloyd's market and regulatory requirements.
Translate business needs into scalable cloud data architectures by partnering with architects, SMEs, product owners, and senior leaders.
Own and resolve complex technical challenges while championing CI/CD automation, cloud‑native best practices, data lifecycle management, and high engineering standards.
Oversee end‑to‑end data engineering delivery, ensuring pipeline reliability, observability, performance, resilience, and continuous improvement.Your Skills:

Expert‑level hands‑on experience with Azure Data Services and Databricks in enterprise, highly regulated environments.
Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation.
Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing CI/CD pipelines (Azure DevOps, branching/merging, automation).
Solid knowledge of data governance, lineage, access controls, secure cloud engineering, and FinOps principles.
Exceptional communication and leadership skills with the ability to influence senior stakeholders and deliver results in fast‑paced, high‑expectation environments.Damia Group Limited acts as an employment agency for permanent recruitment and employment business for the supply of temporary workers. By applying for this job you accept our Data Protection Policy which can be found on our website.

Please note that no terminology in this advert is intended to discriminate on the grounds of a person's gender, marital status, race, religion, colour, age, disability or sexual orientation. Every candidate will be assessed only in accordance with their merits, qualifications and ability to perform the duties of the job.

Damia Group is acting as an Employment Business in relation to this vacancy and in accordance to Conduct Regulations 2003