Senior Data Engineer

Job summary

The Data and Analysis for Delivery (DA4D) division, part of Delivery and Group Corporate Services in Defra, plays a pivotal role in enabling data-driven decision-making across the department’s major delivery portfolio. Within DA4D, the Delivery Data team is responsible for the collection, management, and exploitation of data relating to the major delivery portfolio, performance and risk. By ensuring accurate, timely, and accessible data, the team supports Defra in understanding delivery progress and making informed strategic decisions. Its work underpins the department’s ability to manage complex programmes effectively and achieve outcomes that benefit the departments core purpose.

You will play a key role in driving improvements in how Defra collects, manages, and harnesses data at enterprise level.

Working closely with delivery teams, you will help identify, manage, and unlock the value of critical data assets, enabling evidence-based decision-making and more efficient operations.

Your day-to-day work will be varied and impactful. One day you may be designing and scoping robust data architectures or modern storage solutions to enhance data quality, integrity, and accessibility across the organisation. On another, you might be developing and optimising data pipelines that deliver trusted, high-quality data to data scientists, analysts, and self-service users, supporting their need for timely and accurate insights.

Collaboration will be at the heart of your role. You will work with a wide range of colleagues from across Defra, its arm’s length bodies, and external partners, optimising data flows, integrating with multiple data collection platforms, and ensuring data solutions are secure, scalable, and fit for purpose. Through your work, you will help shape and deliver Defra’s ambition to become a data-driven organisation.

Job description

The team has recently delivered a critical milestone: an end-to-end Delivery Data Service, including a platform for collecting data, pipelines for ingesting and cleaning data, and a data science hub for analysis, visualisation, and interpretation. This foundation enables key services such as Portfolio Management, Enterprise Performance, and Enterprise Risk.

Building on this, you will lead data engineering workstreams that enhance and scale our data architecture, optimise data flows, and deliver new pipelines to bring critical datasets into the team. Your focus will be on designing and implementing robust, cloud-based solutions that ensure data is accurate, accessible, and performant across the organisation. You’ll work with technologies such as Azure Synapse, Databricks, Python, and SQL to develop ETL processes, improve data quality, and enable downstream analytics and reporting.

Collaboration will be central to your role. You’ll partner with colleagues across Defra group to integrate multiple data sources, streamline ingestion processes, and ensure solutions are secure, scalable, and aligned with best practices. This is your opportunity to shape how Defra manages and exploits data at enterprise scale, supporting one of the most complex delivery portfolios in government.

You’ll thrive in a fast-paced, evolving environment - seeing opportunity in ambiguity, adapting quickly, and learning openly. As part of a small, ambitious, and friendly team, you’ll combine technical expertise with creativity to deliver meaningful change.

Key Responsibilities:

• Design, build, and maintain cloud-based data platforms and architectures that integrate large, complex, and disparate datasets, ensuring scalability, security, and performance.
• Develop and optimise ETL pipelines to ingest, transform, and deliver high-quality data for analytics, reporting, and operational use.
• Implement robust data models and storage solutions that support efficient querying, data integrity, and enterprise-wide accessibility.
• Collaborate with stakeholders across Defra group to understand data requirements, integrate multiple sources, and deliver solutions that meet business needs.
• Ensure data quality and governance by applying validation frameworks, metadata management, and lineage tracking to maintain trusted datasets.
• Stay current with emerging data engineering tools and best practices, contributing to innovation and continuous improvement within the team.
• Lead technical workstreams and mentor colleagues, fostering a culture of collaboration, knowledge sharing, and engineering excellence.

Person specification

The ideal candidate will bring deep expertise in cloud-based data solutions, with hands-on experience developing and managing data platforms on technologies such as Azure and Databricks. They will excel at identifying opportunities to reuse and enhance existing workflows, optimising code to drive performance, scalability, and resilience. Equally comfortable engaging with both technical and non-technical stakeholders, they will act as a bridge between engineering teams, data consumers, and business leaders, ensuring data solutions are aligned with organisational goals.

Technical Skills

Essential

• Proven experience designing and building cloud-based data platforms that integrate large, complex, and disparate datasets — ideally using Azure Synapse, Databricks, or similar technologies.
• Advanced programming skills in Python (including PySpark, Pandas) and SQL for data engineering and analytics workloads.
• Strong data modelling expertise, with a deep understanding of industry standards, best practices, and performance optimisation techniques.
• Ability to lead the development of a portfolio of ETL pipelines, across a range of complex and/or large volume data.
• Skilled at defining and implementing robust test conditions and validation frameworks based on business and technical requirements.

Non-technical Skills

Essential

• Outstanding communicator, capable of translating complex technical concepts into clear, actionable insights for technical teams and non-technical stakeholders alike.
• Proactive and forward-looking, with a passion for horizon scanning and keeping up to date with emerging data tools, platforms, and best practices.
• Strong problem-solving abilities — able to investigate complex data challenges, identify root causes, engage the right expertise, and deliver effective, sustainable solutions.
• Strong team player with excellent collaboration skills, able to work effectively with colleagues to achieve shared goals.

Behaviours

We'll assess you against these behaviours during the selection process:

• Changing and Improving
• Communicating and Influencing
• Making Effective Decisions
• Managing a Quality Service

We only ask for evidence of these behaviours on your application form:

• Changing and Improving

Technical skills

We'll assess you against these technical skills during the selection process:

• Ability to demonstrate the skills and knowledge for extracting, transforming, and modelling data. A short exercise will be shared with candidates for them to complete and return ahead of their interview.

Benefits

Alongside your salary of £42,665, Department for Environment, Food and Rural Affairs contributes £12,360 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides.

Government Digital and Data Profession Capability framework skills supplement

This post offers successful candidates the opportunity to be considered for an additional Government Digital and Data Profession Capability framework skills supplement. The level of the awarded supplement is determined following a separate skills assessment process, the outcome of which has no impact on the decision to make a job offer. This digital capability supplement ranges from £0 to £19,696 and you may be awarded a pay supplement within this range based on results of your additional skills assessment.

Technology

We’re a firm believer in agile and flexible working. When you join Defra, we’ll kit you out with a new Windows 10 Laptop, smartphone and rucksack to make sure that you perform your best wherever you are.

Leave

25 days' paid leave per annum rising to 30 days after five years’ service, in addition to eight public holidays and the King’s Birthday privilege day. We also offer

• Competitive maternity, paternity, parental and adoption leave;
• Study leave for the pursuit of academic studies;
• 3 days paid leave for volunteering per annum;
• 15 days paid leave per year for reservist training; and,
• Generous paid sick leave.

Cycle to Work Scheme

Defra loans you a bike for your commute as a tax-free benefit. At the end of the loan period, you have the option to buy the bike, saving 30% of the original cost.

Season Ticket and Bicycle Loan

We offer interest-free season ticket loans, to all Defra employees with more than three months’ service, for purchasing train and bus season tickets, as well as station car parking. For those unable to take advantage of tax-free benefits, we also offer an interest-free bicycle loan to enable the purchase of a bicycle for commuting.

Gym, Social and Sports Activities

Our offices have a range of different onsite facilities, including gym, fitness facilities, games room, showers and cycle storage. There are also a variety of networks and social groups to join. Ask your line manager for more information.

Restaurant and Café Facilities

A range of our offices offer onsite restaurant and café facilities.

Civil Service Benefits

Pension

Your pension is a valuable part of your total reward package, and you will be given the chance to join one of three different pension plans. You can find more information on the Civil Service Pension on the My CSP Website at www.civilservicepensionscheme.org.uk

Civil Service Sports Council

The Civil Service Sports Council (CSSC) organises a wide choice of activities throughout the UK, from art, gliding, abseiling and orienteering to cabaret weekends and quiz and curry nights. Your CSSC membership will provide you with FREE access to hundreds of national and local sites, in addition to savings at many high street stores, cinemas, restaurants and historic houses.

Things you need to know

Artificial intelligence

Artificial intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance (opens in a new window) for more information on appropriate and inappropriate use.

Selection process details

This vacancy is using Success Profiles (opens in a new window), and will assess your Behaviours, Experience and Technical skills.

Application Process

As part of the application process you will be asked to complete a CV, a personal statement and lead behaviour.

Further details around what this will entail are listed on the application form.

Sift

Should a large number of applications be received, an initial sift may be conducted using the Personal Statement. Candidates who pass the initial sift may be progressed to a full sift or progressed straight to assessment/interview.

Interview

If successful at application stage, you will be required to complete a short technical exercise and invited to interview where you will be assessed on Behaviours.

• Behaviour 1 (Lead behaviour) Changing and Improving
• Behaviour 2 Communicating and Influencing
• Behaviour 3 Making Effective Decisions
• Behaviour 4 Managing a Quality Service

Candidates invited to interview will also be sent a technical exercise ahead of their interview. Further information on this exercise will be shared with candidates separately. Interviews are expected to last no longer than 60 Minutes.

Your interview will be conducted by video. Details of how to join your interview will be shared with you by the hiring manager.

Sift and interview dates to be confirmed.

Further Information

As part of the pre-employment process for this post, successful candidate(s) will be able to agree a contractual workplace from those locations listed in this advert. The agreed contractual workplace is then the substantive and permanent place of work for the successful candidate(s)

Where the location is ‘National’ the successful appointee should discuss and agree an appropriate contractual location in line with both Defra’s location policy and site capacity, prior to proceeding with pre-employment processes.

Successful applicants currently employed by the hiring Defra organisation for this post may choose to remain in their current contractual location or may choose to change contractual location to one of those listed above. This should be discussed and agreed prior to proceeding with pre-employment processes.

The agreed amount of time spent at a workplace for this post will reflect the requirement for Civil Servants to spend at least 60% of their working time in an organisation workplace with the option to work the remaining time flexibly from home. Working time spent at a workplace may include time spent at other organisational locations including field-based operational locations, together with supplier, customer or partner locations. This is a non-contractual agreement which is consistent with common Civil Service expectations.

Travel costs to non-contractual workplaces will be subject to departmental travel and subsistence policies. Travel costs to contractual workplaces are the responsibility of the employee.

The successful candidate is required to carry out all their duties from a UK location, and cannot do so from an overseas location at any time.

Defra includes the core department, APHA, RPA, Cefas and VMD.

Salary

New entrants to the Civil Service are expected to start on the minimum of the pay band. The internal roles rules apply to existing Civil Servants, i.e. level transfers move on current salary or the pay range minimum, transfers on promotion move to new pay range minimum or receive 10% increase. Either case is determined by whichever is the highest.

Reserve List

A reserve list may be held for a period of 12 months from which further appointments can be made.

Childcare Vouchers

Any move to Defra from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax Free Childcare; for further information visit the Childcare Choices website.

Reasonable Adjustment

If a person with disabilities is put at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes.

If you need a change to be made so that you can make your application, you should:

Contact Government Recruitment Service via defrarecruitment.grs@cabinetoffice.gov.uk as soon as possible before the closing date to discuss your needs.

Complete the ‘Assistance required’ section in the ‘Additional requirements’ page of your application form to tell us what changes or help you might need further on in the recruitment process. For instance, you may need wheelchair access at interview, or if you’re deaf, a Language Service Professional.

Accessibility

If you are experiencing accessibility problems with any attachments on this advert, please contact the email address in the 'Contact point for applicants' section.

Visa Sponsorship Statement

Please take note that Defra does not hold a UK Visa & Immigration (UKVI) Skilled Worker License sponsor and are unable to sponsor any individuals for Skilled Worker Sponsorship.

Merit Lists

Where more than one location is advertised, candidates will be appointed in merit order for each location. You will be asked to state your location preference on your application.

Criminal Record Check

If successful and transferring from another Government Department a criminal record check may be carried out.

In order to process applications without delay, we will be sending a Criminal Record Check to Disclosure and Barring Service on your behalf.

However, we recognise in exceptional circumstances some candidates will want to send their completed forms direct. If you will be doing this, please advise Government Recruitment Service of your intention by emailing Pre-EmploymentChecks.grs@cabinetoffice.gov.uk stating the job reference number in the subject heading.

Internal Fraud Database Check

Applicants who are successful at interview will be, as part of pre-employment screening, subject to a check on the Internal Fraud Database (IFD). This check will provide information about employees who have been dismissed for fraud or dishonesty offences. This check also applies to employees who resign or otherwise leave before being dismissed for fraud or dishonesty had their employment continued. Any applicant’s details held on the IFD will be refused employment.

A candidate is not eligible to apply for a role within the Civil Service if the application is made within a 5 year period following a dismissal for carrying out internal fraud against government.

For further information on National Security Vetting please visit the Demystifying Vetting website.

Feedback will only be provided if you attend an interview or assessment.

Security

Successful candidates must undergo a criminal record check.Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check (opens in a new window).

See our vetting charter (opens in a new window).People working with government assets must complete baseline personnel security standard (opens in new window) checks.

Nationality requirements

This job is broadly open to the following groups:

• UK nationals
• nationals of the Republic of Ireland
• nationals of Commonwealth countries who have the right to work in the UK
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Further information on nationality requirements (opens in a new window)

Working for the Civil Service

The Civil Service Code (opens in a new window) sets out the standards of behaviour expected of civil servants.

We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window).The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria.The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy.

Diversity and Inclusion

The Civil Service is committed to attract, retain and invest in talent wherever it is found. To learn more please see theCivil Service People Plan (opens in a new window) and the Civil Service Diversity and Inclusion Strategy (opens in a new window).

Apply and further information

This vacancy is part of the Great Place to Work for Veterans (opens in a new window) initiative.The Civil Service welcomes applications from people who have recently left prison or have an unspent conviction. Read more about prison leaver recruitment (opens in new window).Once this job has closed, the job advert will no longer be available. You may want to save a copy for your records.

Contact point for applicants

Job contact :

• Name : Kieran Shields-Merry
• Email : kieran.shields-merry@defra.gov.uk

Recruitment team

• Email : defrarecruitment.grs@cabinetoffice.gov.uk

Further information

If you feel your application has not been treated in accordance with the Recruitment Principles and you wish to make a complaint, in the first instance, you should contact Government Recruitment Services via email: defrarecruitment.grs@cabinetoffice.gov.uk
If you are not satisfied with the response you receive from the Department, you can contact the Civil Service Commission: here