Pyspark Developer

PySpark + Fabric Developer (Contract) | London | Office-Based
Location: London (Office-based)
Contract: 6 months (potential extension)
Start: ASAP
Rate: Market rate - Inside IR35
We’re looking for experienced PySpark + Fabric Developers to join a major transformation programme with a leading global financial data and infrastructure organisation. This is an exciting opportunity to work on cutting-edge data engineering solutions, driving innovation and performance at scale.
Key Responsibilities

• Design, build, and optimise data pipelines for both batch and streaming workloads.
  
• Develop and manage dataflows and semantic models to support analytics and reporting.
  
• Implement complex data transformations, aggregations, and joins with a focus on performance and reliability.
  
• Apply robust data validation, cleansing, and profiling techniques to maintain accuracy.
  
• Enforce role-based access, data masking, and compliance standards.
  
• Tune and optimise workloads to minimise latency and enhance throughput.
  
• Collaborate with analysts and stakeholders to translate business needs into technical solutions.
  
• Maintain clear documentation and contribute to internal best practices.
  Requirements
  
• Strong hands-on experience with PySpark (RDDs, DataFrames, Spark SQL).
  
• Proven ability to build and optimise ETL pipelines and dataflows.
  
• Familiar with Microsoft Fabric or similar lakehouse/data platform environments.
  
• Experience with Git, CI/CD pipelines, and automated deployment.
  
• Knowledge of market data, transactional systems, or financial datasets.
  
• Excellent communication skills and collaborative mindset.
  Desirable
  
• Experience with Azure Data Lake, OneLake, or distributed computing environments.
  
• Understanding of data security and compliance (e.g., GDPR, SOX).
  
• Exposure to preparing datasets for Power BI