Lead Data Engineer - Hadoop - Spark - Python

Sheffield
4 days ago
Create job alert

Job Title: Lead Data Engineer - Hadoop, Spark, Pytthon
Location: Sheffield - 3 days per week in the office
Salary/Rate: Up to £617 per day inside IR35
Start Date: 02/03/2026
Job Type: Contract until November

We are seeking a Lead Data Engineering Consultant with proven experience in leading and developing data engineering platforms.

The ideal candidate will possess hands-on expertise in the following areas:

Extensive enterprise experience with Hadoop, Spark, and Splunk.
Proficiency in object-oriented and functional scripting, particularly in Python.
Skilled in handling raw, structured, semi-structured, and unstructured data (SQL and NoSQL).
Experience integrating large, disparate datasets using modern tools and frameworks.
Strong background in building and optimizing ETL/ELT data pipelines.
Familiarity with source control and implementing Continuous Integration, Delivery, and Deployment via CI/CD pipelines.
Experience supporting and collaborating with BI and Analytics teams in fast-paced environments.
Ability to pair program and work effectively with other engineers.
Excellent analytical and problem-solving abilities.
Knowledge of agile methodologies such as Scrum or Kanban is a plus.
Comfortable representing the team in standups and problem-solving sessions.
Capable of driving the creation of technical test plans and maintaining records, including unit and integration tests, within automated test environments to ensure high code quality.
Promote SRE (Site Reliability Engineering) culture by addressing challenges through data engineering.
Ensure service resilience, sustainability, and adherence to recovery time objectives for all delivered software solutions.If you are interested in this opportunity, please apply now with your updated CV in Microsoft Word/PDF format.

Disclaimer
Notwithstanding any guidelines given to level of experience sought, we will consider candidates from outside this range if they can demonstrate the necessary competencies.
Square One is acting as both an employment agency and an employment business, and is an equal opportunities recruitment business. Square One embraces diversity and will treat everyone equally. Please see our website for our full diversity statement

Related Jobs

View all jobs

Lead Data Engineer

Lead Data Engineer

Lead Data Engineer

Lead Data Engineer - Azure Synapse

Lead Data Platform Engineer | Remote | £85k + 4 day week

Lead Data Platform Engineer | Remote | £85k + 4 day week

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

How Many Cloud Computing Tools Do You Need to Know to Get a Cloud Job?

If you are aiming for a role in cloud computing, it can feel like the skills list never ends. One job advert asks for AWS, Terraform and Kubernetes. Another mentions Azure DevOps, PowerShell and ARM templates. A third throws in Docker, Python, Linux, CI/CD, monitoring tools and security frameworks. It is no surprise that many cloud job seekers feel overwhelmed before they even apply. Here is the reality most cloud hiring managers agree on: they are not hiring you because you know every cloud tool. They are hiring you because you understand cloud concepts, can design reliable systems, manage costs, keep things secure and support real workloads. Tools matter, but only when they support outcomes. So how many cloud computing tools do you actually need to know to get a job? For most roles, the answer is far fewer than you think. This article explains what employers really expect, which tools are essential, which are role-specific, and how to focus your learning so you look capable and employable rather than scattered.

What Hiring Managers Look for First in Cloud Computing Job Applications (UK Guide)

anding a job in cloud computing can be highly competitive — especially in the UK market where demand far outpaces supply in many segments. Whether you’re aiming for roles in Cloud Engineering, DevOps, Site Reliability, Cloud Architecture, Security, Data/Analytics, or Platform Operations, hiring managers screen applications quickly and with specific priorities in mind. Hiring managers don’t read every detail at first; they scan for critical signals in the first 10–20 seconds. These early signals determine whether your CV gets read more closely, whether your LinkedIn profile gets clicked, and whether you’re invited to interview. This guide breaks down, in practical terms, exactly what hiring managers look for first in cloud computing applications — and what you should emphasise in your CV, cover letter and portfolio to stand out on www.cloudcomputingjobs.co.uk .

The Skills Gap in Cloud Computing Jobs: What Universities Aren’t Teaching

Cloud computing underpins almost every modern digital service. From financial systems and healthcare platforms to AI, e-commerce, government infrastructure and cybersecurity, the cloud is now the default operating environment for UK organisations. Demand for cloud professionals has grown rapidly, with roles spanning architecture, engineering, security, DevOps, platform operations and cost optimisation. Salaries remain high, and vacancies remain stubbornly difficult to fill. Yet despite a growing number of graduates with computer science, IT and software engineering degrees, employers across the UK report a persistent problem: Too many candidates are not job-ready for real cloud computing roles. This is not a question of intelligence or motivation. It is a structural skills gap between what universities teach and what cloud jobs actually require. This article explores that gap in depth: what universities do well, what they consistently miss, why the gap exists, what employers genuinely want, and how jobseekers can bridge the divide to build sustainable careers in cloud computing.