Data Engineer

I'm seeking a Data Engineer with extensive hands-on experience in Databricks, Azure Data Factory, Python, and DBT. The suitable candidate will develop data models, troubleshoot pipelines, and deliver production-ready code. Effective communication skills and the ability to articulate technical decisions clearly are essential for this role. Additionally, you will be responsible for automating ETL workflows that feed into machine learning features, enhancing the efficiency and accuracy of ML models.
Responsibilities:

• Develop and maintain SQL models and macros using DBT
  
• Collaborate with stakeholders to gather and refine modelling requirements
  
• Debug and enhance existing data models
  
• Design and implement ETL/ELT workflows and data pipelines using Azure Data Factory
  
• Automate ETL workflows to support machine learning features
  
• Apply best practices for data governance, quality, and performance
  
• Utilise Azure Databricks and adhere to code-based deployment practices
  Essential Skills:
  
• Over 3 years of experience with Databricks (including Lakehouse, Delta Lake, PySpark, Spark SQL)
  
• Strong proficiency in SQL with 5+ years of experience
  
• Extensive experience with Azure Data Factory
  
• Proficiency in Python programming
  
• Excellent stakeholder / client-facing communication abilities
  Desirable Skills:
  
• Databricks certification
  
• Experience with Terraform for infrastructure as code
  
• Background in financial services or insurance industries
  
• Understanding of data governance frameworks
  
• Knowledge of best practices in data security
  If you are interested, apply with your CV and I will contact you to discuss