Senior Data Engineer - Python, Databricks & Counterparty Credit Risk - Venn Group

Senior Data Engineer – Python, Databricks & Counterparty Credit Risk

Location: London

Working Pattern: Hybrid – 2–3 days per week onsite

We are recruiting for an experiencedSenior Data Engineer to join a leading global banking organisation, working across a highly technical data environment supportingCounterparty Credit Risk (CCR), Credit Risk and wider risk data .

This is a hands-on engineering position requiring strong expertise across Python, PySpark, SQL and Databricks , alongside experience designing, building and optimising scalable data pipelines and Lakehouse solutions.

Candidates should ideally have previously worked withCounterparty Credit Risk or Credit Risk data , including areas such as counterparty/exposure data, risk analytics, regulatory risk or associated banking risk datasets.

Key Responsibilities

1. Design, develop and maintain scalable data pipelines usingPython, PySpark and Databricks
2. Build and optimiseETL/ELT workflows using Spark and Delta Lake
3. Engineer and transform data supportingCounterparty Credit Risk, Credit Risk and wider risk functions
4. Work with datasets relating to areas such ascounterparties, credit exposures, limits, collateral and risk analytics
5. Design robust data models and architectures acrossLakehouse / Medallion environments
6. Develop and manage Databricks notebooks, jobs and workflows
7. Optimise large-scale distributed data processing and troubleshoot Spark performance
8. Implement strongdata quality, reconciliation, governance and monitoring controls
9. Build reliable logging and alerting across production data pipelines
10. Integrate data platforms withPower BI dashboards and wider applications
11. Support CI/CD and engineering best practices across development, testing and production environments
12. Work closely with Risk, Technology, Analytics and other senior stakeholders

Required Experience

1. 10+ years' experience across Data Engineering or closely related data disciplines
2. Strong hands-onPython development experience
3. StrongPySpark / Apache Spark engineering experience
4. Extensive hands-on experience withDatabricks , including notebooks, jobs/workflows and Delta Lake
5. AdvancedSQL and relational database experience
6. Strong track record designing and building scalableETL/ELT data pipelines
7. Experience withDelta Lake, Lakehouse and/or Medallion architectures
8. Experience handlingbanking risk data , ideally within Counterparty Credit Risk (CCR), Credit Risk, counterparty exposure or closely related risk domains
9. Strong understanding of data modelling, data quality and production data engineering
10. Cloud experience acrossAzure, AWS and/or GCP
11. Experience with modern data platforms/warehouses such as Snowflake, BigQuery or Redshift
12. Experience integrating data platforms withPower BI
13. Experience with orchestration such asDatabricks Workflows or Airflow

Highly Desirable

1. Direct experience engineering data forCounterparty Credit Risk , including exposure/counterparty datasets
2. Knowledge of areas such asPFE, EPE, EAD, CVA/XVA, SA-CCR, collateral, netting or counterparty limits
3. Previous experience within aninvestment bank, global bank or capital-markets environment
4. Databricks Data Engineer certification
5. Kafka / Structured Streaming experience
6. Delta LakeCDC, Time Travel and optimisation
7. CI/CD, Git and DevOps experience
8. Docker / Kubernetes experience
9. Experience exposing data through APIs or supportingReact-based front-end applications
10. Experience supporting analytics or ML workloads within Databricks

We are particularly interested ingenuinely hands-on Senior Data Engineers who can demonstrate the Databricks and Python solutions they have personally designed, built, optimised and supported in production.

If you have strongDatabricks, Python and PySpark engineering experience alongside exposure toCounterparty Credit Risk, Credit Risk or banking risk data , please apply for further information.