Azure Databricks Engineer

Huxley Associates
City of London, United Kingdom
2 weeks ago
Posted
9 Apr 2026 (2 weeks ago)

This is a rare opportunity to apply serious data engineering in a domain where latency, correctness, and reliability carry direct commercial weight.

* Requirements

* 6+ years data engineering in production environments; Python expertise - idiomatic, well-tested, production-grade code, not notebook scripts

* ETL/ELT pipeline design and implementation at scale; orchestration with Airflow, Prefect, or equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics

* Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage; infrastructure as code for data workloads (Terraform or Bicep)

* Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs, cost-aware compute management, Spark job optimisation

* Relational databases: PostgreSQL at production scale - query optimisation, indexing strategies, table partitioning, replication, schema design for both OLTP and analytical workloads

* MongoDB - document modelling, aggregation pipelines, indexing strategy, replica sets; clear judgment on when document vs relational storage is the right architectural call

* Containerisation: Docker and Kubernetes-based deployment of data workloads; reproducible, environment-agnostic data infrastructure

* Data modelling for analytical workloads - dimensional modelling, data vault, or equivalent; schema evolution, slowly changing dimensions, and downstream impact analysis

* Stream and batch processing patterns; late data handling, watermarking, and backfill strategies; throughput vs latency trade-offs in pipeline design

* Production data observability - data lineage, quality checks, SLA monitoring, alerting on freshness and completeness; treating data correctness as a first-class concern

* CI/CD for data infrastructure - version-controlled pipelines, automated data quality testing, reproducible and auditable deploys

* Ability to work directly with quant researchers, risk managers, and traders - translate business requirements into reliable, well-documented data products

* Nice to Have

* Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management

* Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem

* dbt or equivalent SQL transformation layer; experience building and maintaining dbt projects in a production data warehouse

* Event streaming with Kafka or Confluent Platform - topic design, consumer group management, exactly-once delivery guarantees

* OLAP-optimised stores - ClickHouse, DuckDB, or equivalent; understanding of columnar storage and vectorised query execution

* Energy, commodities, or broader financial markets domain knowledge

* What We're Looking For

* You treat data as a product, not a side effect. You know what it takes to make a pipeline trustworthy - not just running, but observable, tested, and recoverable when something upstream changes at 3am. You think in systems: schema evolution, lineage, freshness SLAs, and the downstream impact of every modelling decision. At ETrading , that data is the foundation of billion-dollar trading decisions. You are the reason it is right.

To find out more about Huxley, please visit

Huxley, a trading division of SThree Partnership LLP is acting as an Employment Business in relation to this vacancy | Registered office | 8 Bishopsgate, London, EC2N 4BQ, United Kingdom | Partnership Number | OC(phone number removed) England and Wales

Related Jobs

View all jobs

Lead Platform Engineer

Synapri London, United Kingdom

Data Architect - Bristol Opportunity

Hays Technology Bristol, United Kingdom
Hybrid

Data Platform Solutions Architect (Professional Services)

Databricks London, United Kingdom
£40,000 – £80,000 pa Hybrid

Resident Solutions Architect (Professional Services)

Databricks London, United Kingdom

Data Platform Solutions Architect (Professional Services) - Emerging Enterprise & DNB

Databricks London, United Kingdom

Azure DevOps Engineer, , Midlands

BMR Associates Birmingham, United Kingdom
Permanent

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Where to Advertise Cloud Computing Jobs in the UK (2026 Guide)

Advertising cloud computing jobs in the UK requires a different approach to most technical hiring. The candidate pool is large relative to other deep tech disciplines but highly segmented — cloud architects, DevOps engineers, platform engineers, FinOps specialists and cloud security professionals each occupy distinct communities with different job search behaviours, certification profiles and salary expectations. General job boards reach a broad audience but struggle to differentiate between these disciplines, producing high application volumes but low candidate quality for specialist cloud roles. This guide, published by CloudComputingJobs.co.uk, covers where to advertise cloud computing roles in the UK in 2026, how the main platforms compare, what employers should expect to pay, and what the data says about hiring across different role types.

New Cloud Computing Employers to Watch in 2026: UK and Global Companies Powering the Digital Economy

Cloud computing is no longer just a backbone technology—it is now the engine of digital transformation, underpinning everything from AI and fintech to healthcare and government services. For professionals browsing CloudComputingJobs.co.uk, the biggest opportunities lie with new and fast-scaling employers that are investing heavily in infrastructure, platforms, and next-generation cloud services. In this article, we explore the new cloud computing employers to watch in 2026, focusing on UK-based startups, scale-ups, and global companies expanding their footprint across Britain. These organisations have recently secured funding, launched major projects, or won strategic contracts—clear signals of hiring growth.

Cloud Engineer Jobs in the UK: Salary, Skills, Career Paths & How to Get Hired

Cloud engineer jobs are among the fastest-growing technology roles in the UK. As organisations move infrastructure, applications and data into the cloud, demand for skilled cloud professionals continues to surge across finance, healthcare, retail, defence, government and high-growth startups. If you’re exploring a career in cloud engineering — or looking for your next role — this guide covers everything you need to know: What a cloud engineer does Types of cloud engineer jobs Required skills and certifications UK salary expectations Career progression pathways How to land a cloud engineer job in the UK Whether you’re a graduate, IT professional transitioning into cloud, or an experienced engineer looking to specialise, this article will help you position yourself competitively.