Lead Data Engineer

Waracle are looking for a Lead Data Engineer for an exciting new role joining our world-class digital technology consultancy and home to a diverse, smart, curious and ambitious community of specialists in technology-driven transformation.

We work with ambitious clients to help them solve their biggest business and customer challenges. We help our clients to innovate and create intelligent digital products and services. We thrive on complex  challenges and deliver business-critical IT transformation projects, moving seamlessly from strategy, design and delivery to operations.

This is a Hybrid role working from any of our offices (Glasgow, London, Edinburgh or Dundee)  2 days a week 

As a Lead Data Engineer, you will be a critical technical leader, accountable for the technical delivery and overall quality of large-scale data and AI projects. You will act as a consultant, setting the technical direction to meet complex client requirements and translating business priorities into actionable technical strategy. This role requires a strong balance of technical expertise (ensuring performance, cost, and security are balanced in system designs) and leadership (coaching, conflict de-escalation, and driving improvements across multiple teams). You will ensure technical decisions consistently align with client business objectives, making proactive tradeoffs on scope or resources to maintain delivery pace

Key aspects of the role include

• Owning and directing technical delivery for large-scale data projects, maintaining accountability for technical outcomes, solution quality, and proactively making tradeoffs on scope or resources to maintain delivery pace.

• Defining technical strategy by translating client priorities into actionable plans, and ensuring team delivery consistently aligns with business objectives.

• Designing robust data systems while effectively balancing performance, cost, and security requirements.

• Driving best practices by shaping CI/CD and IaC practices across multiple teams, and defining coding, testing, and review standards across the Data & AI practice.

• Resolving complex challenges including sophisticated integration issues across cloud and on-prem systems, and leading root cause analysis and systemic fixes for major incidents.

• Partnering and aligning stakeholders by working with Product/Delivery to align technical decisions with client outcomes, managing expectations on scope and budget, and de-escalating conflicts.

• Leading project execution by facilitating backlog refinement and workshops, guiding the team in tracking delivery metrics (e.g., DORA), and ensuring clear, up-to-date technical documentation is created.

• Leading and coaching the team by ensuring a collaborative, psychologically safe environment, coaching team members regularly, and formally/informally mentoring others in the practice.

• Upholding Waracle’s values on client engagements and driving cross-project improvements in delivery tooling.

What you'll bring

• Proven track record of designing, implementing, and optimising complex, large-scale data pipelines and cloud infrastructure

• Deep expertise in at least one major cloud platform (AWS, Azure, or GCP), with working knowledge of a second platform preferred

• Advanced programming skills in Python and SQL, with experience in Scala or Java

• Expert-level knowledge of Infrastructure as Code tooling (Terraform, CloudFormation, ARM templates)

• Comprehensive experience with data processing frameworks (Apache Spark, dbt) and orchestration tools (Apache Airflow, Step Functions)

• Strong command of CI/CD tools and methodologies (GitHub Actions, Azure Pipelines, Cloud Build)

• Demonstrable expertise in system design, technical planning, and architectural guidance for distributed data systems

• Exceptional client-facing and consulting abilities, with strong presentation, stakeholder management, and mentoring skills

• Strong understanding and practical application of Agile methodologies (Scrum, Kanban)

The recruitment process you can expect for this role is an initial call with your dedicated Talent Acquisition Partner who will chat with you about Waracle, what you are looking for in a new position, the salary for the role, notice period and benefits (the important stuff!).

After that, you'll be invited to a two-stage interview process where you have an opportunity to find out more about the role and showcase your skills and experience. Your Talent Acquisition Partner will guide you through the whole process to your first day with us.

Our Benefits

We believe in supporting our team, inside and outside of work. Here's a glimpse of what you can expect:

• Rest & Recharge: You'll receive 27 days of annual leave, plus 8 bank holidays, totaling 35 days of holiday to unwind and pursue your passions.

• Financial Future & Security: Plan for retirement with our company-matched 5% pension contribution. For added peace of mind, we also offer a Death in Service benefit, providing 2 x your annual salary for your loved ones.

• Health & Wellness: Our Medicash Health Cash Plans help cover everyday healthcare costs, complemented by an Employee Assistance Programme for confidential support, and Group Sickness Cover for peace of mind.

• Flexible Working: We champion hybrid working, providing a home office setup budget to ensure you're comfortable and productive wherever you are.

• Learning & Development: Invest in yourself with access to Udemy Business and a dedicated learning and development budget for continuous growth.

• Lifestyle Perks: Make the most of our Cycle to Work Scheme, and enjoy exclusive gym and retail discounts.

• Family First: Our enhanced parental leave policies include support for fertility journeys, recognising diverse family needs.

• Community & Connection: We foster a vibrant culture with regular fun meet-ups and lunch on us in the office once a month. You'll also have the opportunity to make a local impact through Spirit of Waracle, our initiative for getting involved in charitable causes locally, building strong team bonds and giving back.

We are an equal opportunities employer and welcome applications from all suitably qualified persons regardless of their race, gender, disability, religion/belief, sexual orientation or age.