Lead Data Engineer

Job summary

At DSIT we’re all about improving people’s lives by maximising the potential of science & technology.

We accelerate innovation, investment and productivity through world-class science, research and development.

We use technology for good by ensuring new and existing technologies are safely developed and deployed across the UK, with the benefits more widely shared.

We are driving forward a modern digital government which gives citizens a more satisfying experience and their time back.

We do all this to enable the Government’s 5 national missions: kickstarting economic growth, making Britain a clean energy superpower, taking back our streets, breaking down barriers to opportunity and building an NHS fit for the future.

Above all, we focus on improving people’s lives. Whether it’s researching new treatments for disease, developing better batteries, reducing burdens through better public services, keeping children safe online, and much more, outcomes for citizens are at the heart of what we do.

Our Inclusive Environment

We offer flexible working benefits, employee well-being support and a great pension. We are enormously proud to be a Disability Confident Leader employer. We support candidates with adjustments throughout our recruitment process. Information about disability confidence and just some examples of the adjustments that you can request can be found in the reasonable adjustment section below.

The Civil Service is committed to attract, retain and invest in talent wherever it is found. To learn more please see the Civil Service People Plan and the Civil Service D&I Strategy.

Find Out More

We regularly run events where you can find out more about the department and tips for the application and interview process. You can sign up for upcoming events here: https://forms.office.com/e/Jae3B4w7xm

You can also follow our LinkedIn Page: https://www.linkedin.com/showcase/dsitcareers/

Job description

This role is central to DSIT’s ambition to unlock the full value of departmental data by replacing fragmented systems with a unified analytical tooling and data platform. By helping deliver this platform, you will directly enable analysts across the department to access tools and data quickly and securely. This will allow them to improve the evidence-base for key AI, science and emerging technologies policy decisions. This work will also provide the foundation for future AI tools, automation, and streamlined cross-team collaboration in DSIT.

The new platform will ensure analysts have:

• Timely, reliable access to high-value datasets
• Tools to carry out reproducible, secure, and efficient analysis
• Clear processes for procuring, ingesting, and sharing data
• A shared environment that reduces duplication and enables joined-up insights across teams.

Our design draws on the Ministry of Justice’s open-source Analytical Platform, widely recognised as a leading model across government, and has been adapted to meet DSIT’s needs as the digital centre of government.

A platform in two parts

The DSIT analytical platform is being delivered through two interconnected strands:

1. Analytical tooling – delivering the secure, scalable tools analysts need, such as JupyterLab, RStudio, VS Code, Git integration, and cloud-based workflow orchestration.
2. Data platform – delivering the pipelines, storage, and ingestion processes that ensure analysts can find and use trusted data efficiently.

Initially, this role will focus on strengthening and extending the Data platform, to meet the needs of analysts in DSIT. As the platform matures, you’ll also have opportunities to tackle broader Data Engineering challenges across the department, including data science and AI applications that improve operational efficiency and widen access to data for non-technical users.

Overall, the technical side of the platform team consists of two Lead DevOps Engineers, two Lead Data Engineers and three Software Developers. The team is led by a Product Owner, with support from a Technical Lead.

Timelines for the platform:

Phase 1: Establishing the core data architecture for ingestion, transformation and storage, setting the foundation for a scalable cloud-based platform that supports a wide range of data sources and use cases. This will already be complete.

Phase 2: User onboarding and refinement. You will join the project at this stage, helping to onboard a core group of analytical teams as early adopters. This will ensure they can easily access and use the data they need. Their feedback and insights will guide further improvements to the data architecture and help us prepare for a wider rollout across the department, including supporting teams with the migration of their existing datasets.

Where this role sits

The role is part of DSIT’s central Data Science and Engineering team, part of the department’s Analysis function. This team plays a key role in shaping DSIT’s analytical capability and works closely with technical colleagues in the Integrated Corporate Services (ICS) Digital team. By sitting within Analysis, the team ensures the platform is developed with analysts’ needs front and centre—owned by the analytical community and built to serve it.

As a G7 Data Engineer working on the Data platform, you will develop and extend the data infrastructure that underpins the entire analytical platform. You’ll establish scalable pipelines and processes, aligned with departmental data governance standards to make data assets accessible and reusable across analytical teams. Your work will contribute to the long-term vision of a single DSIT data system, ensuring the whole organisation is aligned on one trusted data source. This will directly improve evidence-based policymaking, unlock efficiency savings across the department, and create the foundations for future innovation and AI-enabled tools.

Person specification

• Design and deliver data pipelines to ingest, transform, and standardise datasets, to ensure discoverability across the department.
• Use cloud-native tools (e.g. AWS Glue, Lambda, Athena, S3) to develop scalable, maintainable data flows.
• Work closely with the other digital roles including DevOps Engineers and Software Developers to ensure effective delivery.
• Improve and maintain a centralised data store to enable joined-up analysis, reducing duplication and silos.
• Ensure pipelines are well-documented, tested, monitored and able to support user feedback and iteration.
• Work directly with analysts to understand their needs and translate them into robust data solutions.

Essential Criteria

• Deep experience in cloud-based data engineering, ideally with AWS services.
• Strong Python skills for data processing (pandas, pyarrow, duckdb).
• Experience with data orchestration and pipeline frameworks (Airflow, dbt, AWS Glue Workflows).
• Sound knowledge of data modelling, standardisation and best practices in reproducible analytics.
• The ability to work closely with analysts, digital teams and fellow engineers to shape scalable, user-driven data infrastructure.

Desirable Criteria

• Familiarity with analytical tools (e.g. JupyterLab, RStudio, VS Code) and how analysts use them.

Technical skills

We'll assess you against these technical skills during the selection process:

• Communicating between technical and non-technical (practitioner)
• Data development process (expert)
• Programming and build (data and analytics engineering) (practitioner)

Benefits

Alongside your salary of £54,415, Department for Science, Innovation & Technology contributes £15,764 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides.

The Department for Science, Innovation and Technology offers a competitive mix of benefits including:

• A culture of flexible working, such as job sharing, homeworking and compressed hours.
• Automatic enrolment into the Civil Service Pension Scheme, with an employer contribution of 28.97%.
• A minimum of 25 days of paid annual leave, increasing by 1 day per year up to a maximum of 30.
• An extensive range of learning & professional development opportunities, which all staff are actively encouraged to pursue.
• Access to a range of retail, travel and lifestyle employee discounts.

Office attendance

The Department operates a discretionary hybrid working policy, which provides for a combination of working hours from your place of work and from your home in the UK. The current expectation for staff is to attend the office or non-home based location for 40-60% of the time over the accounting period.

Things you need to know

Artificial intelligence

Artificial intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance (opens in a new window) for more information on appropriate and inappropriate use.

Selection process details

This vacancy is using Success Profiles (opens in a new window), and will assess your Experience and Technical skills.

As part of the application process you will be asked to complete a CV and personal statement.

Further details around what this will entail are listed on the application form.

Please use your personal statement (in no more than 750 words) to describe how you meet the essential criteria for this role.

Applications will be sifted on CV and Personal Statement.

In the event of a large number of applicants, applications will be sifted on the CV only.

Candidates who pass the initial sift may be progressed to a full sift, or progressed straight to assessment/interview.

The interview will consist of technical questions.

Interviewees will be asked to deliver a presentation; further details will be provided nearer the time.

The link to the technical/professional competency framework that candidates will be assessed against, for their reference: Data engineer - Government Digital and Data Profession Capability Framework

Sift and interview dates

Sift and interview dates to be confirmed.

Further Information

For further information on National Security Vetting please visit the Demystifying Vetting website.

Existing Civil Servants and applicants from accredited NDPBs are eligible to apply,and can be considered on loan basis (Civil Servants) or secondment (accredited NDPBs). Prior agreement to be released on a loan basis must be obtained before commencing the application process. In the case of Civil Servants, the terms of the loan will be agreed between the home and host department and the Civil Servant. This includes grade on return.

Reasonable Adjustment

We are proud to be a disability confident leader and we welcome applications from disabled candidates and candidates with long-term conditions.

Information about the Disability Confident Scheme (DCS) and some examples of adjustments that we offer to disabled candidates and candidates with long-term health conditions during our recruitment process can be found in our DSIT Candidate Guidance. A DSIT Plain Text Version of the guidance is also available.

We encourage candidates to discuss their adjustment needs by emailing the job contact which can be found under the contact point for applicants section.

If you are experiencing accessibility problems with any attachments on this advert, please contact the email address in the 'Contact point for applicants' section.

If successful and transferring from another Government Department a criminal record check may be carried out.

New entrants are expected to join on the minimum of the pay band.

A location based reserve list of successful candidates will be kept for 12 months. Should another role become available within that period you may be offered this position.

Candidates who meet the minimum benchmark may be placed on a Reserve List for consideration for similar roles, including those at a lower grade. Candidates who narrowly miss the benchmark and are not placed on the Reserve List may still be considered for an offer in a similar role at a lower grade.

Please note terms and conditions are attached. Please take time to read the document to determine how these may affect you.

Any move to the Department for Science, Innovation and Technology from another employer will mean you can no longer access childcare vouchers. This includes moves between government departments. You may however be eligible for other government schemes, including Tax Free Childcare; for further information visit the Childcare Choices website.

DSIT does not normally offer full home working (i.e. working at home); but we do offer a variety of flexible working options (including occasionally working from home).

DSIT cannot offer Visa sponsorship to candidates through this campaign.

DSIT holds a Visa sponsorship licence but this can only be used for certain roles and this campaign does not qualify.

In order to process applications without delay, we will be sending a Criminal Record Check to Disclosure and Barring Service /Disclosure Scotland on your behalf.

However, we recognise in exceptional circumstances some candidates will want to send their completed forms direct. If you will be doing this, please advise Government Recruitment Service of your intention by emailing Pre-EmploymentChecks.grs@cabinetoffice.gov.uk stating the job reference number in the subject heading.

For further information on the Disclosure Scotland confidential checking service telephone: the Disclosure Scotland Helpline on 0870 609 6006 and ask to speak to the operations manager in confidence, or email Info@disclosurescotland.co.uk

Applicants who are successful at interview will be, as part of pre-employment screening, subject to a check on the Internal Fraud Database (IFD). This check will provide information about employees who have been dismissed for fraud or dishonesty offences. This check also applies to employees who resign or otherwise leave before being dismissed for fraud or dishonesty had their employment continued. Any applicant’s details held on the IFD will be refused employment.

A candidate is not eligible to apply for a role within the Civil Service if the application is made within a 5 year period following a dismissal for carrying out internal fraud against government.

Feedback

Feedback will only be provided if you attend an interview or assessment.

Security

Successful candidates must undergo a criminal record check.Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check (opens in a new window).

See our vetting charter (opens in a new window).People working with government assets must complete baseline personnel security standard (opens in new window) checks.

Nationality requirements

This job is broadly open to the following groups:

• UK nationals
• nationals of the Republic of Ireland
• nationals of Commonwealth countries who have the right to work in the UK
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Further information on nationality requirements (opens in a new window)

Working for the Civil Service

The Civil Service Code (opens in a new window) sets out the standards of behaviour expected of civil servants.

We recruit by merit on the basis of fair and open competition, as outlined in the Civil Service Commission's recruitment principles (opens in a new window).The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria.The Civil Service also offers a Redeployment Interview Scheme to civil servants who are at risk of redundancy, and who meet the minimum requirements for the advertised vacancy.

Diversity and Inclusion

The Civil Service is committed to attract, retain and invest in talent wherever it is found. To learn more please see theCivil Service People Plan (opens in a new window) and the Civil Service Diversity and Inclusion Strategy (opens in a new window).

Apply and further information

This vacancy is part of the Great Place to Work for Veterans (opens in a new window) initiative.The Civil Service welcomes applications from people who have recently left prison or have an unspent conviction. Read more about prison leaver recruitment (opens in new window).Once this job has closed, the job advert will no longer be available. You may want to save a copy for your records.

Contact point for applicants

Job contact :

• Name : Rachel Dunne
• Email : Rachel.dunne2@dsit.gov.uk

Recruitment team

• Email : dsitresourcing.grs@cabinetoffice.gov.uk

Further information

Appointment to the Civil Service is governed by the Civil Service Commission’s Recruitment Principles. If you feel that your application has not been treated in accordance with the recruitment principles, and wish to make a complaint, then you should contact in the first instance DSITrecruitment.grs@cabinetoffice.gov.uk . If you are not satisfied with the response that you receive, then you can contact the Civil Service Commission. For further information on bringing a complaint to the Civil Service Commission please visit their web pages: Click here to visit Civil Service Commission/Complaints.

Attachments

DSIT T&Cs v1.2 Opens in new window (docx, 179kB)