Data Engineer

Our client, a leading global SaaS organisation is looking for a Data Engineer to join their International Engineering team.

This is a great opportunity to contribute greatly to an extremely engineering-focused organization and further your experience across technologies such as Cloud Composer, Airflow, Dataflow, Spark, Terraform, Google Cloud Platform, etc;

You will own engineering efforts to provide partner teams within the organisation with data sources that build the AI/ML user experience. This a contract role 12 months Hybrid role with the opportunity for the role go permanent.

What You'll Do:

* Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;

* Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;

* Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;

* Participate in code and technical design reviews;

* Scope and plan work together with product engineering, data science and the business; support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.

Minimum requirements:

* 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;

* Working knowledge of Python, SQL and shell scripting;

* Experience with Docker/k8s and “infrastructure as a code” tools such as Terraform, Salt or similar;

* 3+ years of significant professional experience developing applications written in a modern programming environment - preferably Scala/Java and/or Python;

* Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;

* Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.

Benefits

* Hybrid working

* Great location with fantastic offices

* Award winning work environment