Data Engineering & Tech Lead

Job summary

The Infected Blood Compensation Authority (IBCA) is a new arm’s-length body set up, at unprecedented pace, to administer compensation to people whose lives have been impacted by the infected blood scandal.

IBCA will ensure payment is made in recognition of the wrongs experienced by those who have been infected by HIV, Hepatitis B or C, as well as those who love and care for them. They have been frustrated and distressed by the delays in achieving proper recognition, and we must help put this right.

We are committed to putting the infected and affected blood community at the centre of every decision we make and every step we take to build our organisation to deliver compensation payments.

IBCA employees will be public servants. If successful in this role you will be appointed directly into IBCA, on IBCA terms and conditions as a public servant.

Successful applicants will join the Civil Service Pension Scheme.

Please note that the mission of IBCA means that it is likely to be operational for a period of approximately 5 to 7 years. When IBCA’s work begins to wind down, IBCA employees will receive support and practical guidance to find a new role, whether in the Civil Service, another Arms Length Body (ALB), or an external employer.

Job description

The Infected Blood Compensation Authority (IBCA) is responsible for delivering a compensation scheme that has been long awaited by the infected blood community to provide financial compensation to victims of infected blood on a UK-wide basis.
This role sits within the Data Operations arm of the IBCA Data Directorate. Working in close collaboration with the Principal Data & Systems Architect, you will be responsible for turning architectural blueprints into a hardened, high-performance data ecosystem.
The Data Operations team is responsible for developing and running safe and secure data solutions that provide a single source of truth for those going through their compensation journey. They are building a new data platform using Amazon Web Services (AWS) and data management and intelligence products using Databricks and Quantexa.
We are taking a product-centric approach treating data as a product and are building squads around our products, with a focus on paying compensation to those impacted by the infected blood scandal seamlessly.
As the Data Engineering & Tech Lead, you will spearhead the engineering execution—designing and implementing robust, automated data pipelines and Master Data Management (MDM) processes. You will ensure code quality, drive a rigorous culture of Test-Driven Development (TDD) and Data Observability, and ensure our engineering practices are scalable and secure by design.
Working at IBCA gives you a huge opportunity to make an impact on those who deserve compensation, and this role suits a candidate who can spearhead solutions from the ground up to take from ideation to reality so that data is an enabler to everything IBCA does.

As the Principal Data Engineering & Tech Lead, you will work in close partnership with the Principal Data & Systems Architect, you will translate high-level designs into high-performance, production-ready code. You are responsible for the "how"—driving the technical standards, framework development, and engineering discipline required to deliver a world-class data platform built on AWS, Databricks, and Quantexa.
You will:
• Set the technical direction for Data, Analytics, Data Test and Data DevOps engineering. You will lead the development of robust, automated data pipelines and master data management processes, ensuring that architectural visions are operationalised using DevSecOps and DataOps best practices.
• Ensure our engineering stack is scalable, secure, and efficient. You will develop reusable engineering frameworks and libraries that accelerate delivery across all squads while maintaining strict quality gates.
• Lead the recruitment and development of our data engineering teams. You will shape career paths and L&D routeways, ensuring our engineers have the skills to handle high-complexity tasks like Spark optimisation and entity resolution.
• Provide direct leadership to engineering squads. You will perform deep-dive code reviews, and unblock high-level technical hurdles that require expert-level intervention.
• Work across digital service teams and with business stakeholders to translate complex requirements into concrete technical roadmaps and Quality Assurance (QAT) approaches that are feasible and performant.
• Collaborate with industry partners (e.g. Databricks, Quantexa) to influence their product roadmaps and leverage new features that give IBCA a technical edge.

• Engineering Implementation Lead: Partner with the Principal Data & Systems Architect to operationalise architectural patterns. You will lead technical execution on the ground, ensuring that data pipelines and data platform features are built to be resilient, reusable, and performant.
• Engineering Standards & Frameworks: Work in partnership with the Chief Technology Officer and the wider IBCA engineering community to define and enforce the "gold standard" for code quality. You will develop shared engineering libraries, common ETL/ELT frameworks, and standardized CI/CD patterns to accelerate delivery across all data squads.
• Deep Technical Mentorship: You will spend significant time performing deep-dive code reviews, pair programming on complex logic (e.g. Spark optimisation or Quantexa entity resolution), and unblocking technical hurdles in delivery teams.
• Quality & Reliability Engineering: Lead the transition from basic testing to data observability. You will implement sophisticated automated testing, data quality monitoring, and "circuit breaker" patterns to ensure platform reliability and trust.
• DevSecOps & Automation: Take ownership of the deployment lifecycle. You will lead the automation of our infrastructure (IaC) and data deployments, ensuring that security "shift-left" principles are embedded directly into the engineering workflow.
• Technical Resource & Capability Building: Shape the data engineering culture. You will identify skill gaps, design internal L&D "routeways" for Data Engineers, Analytical Engineers, Data Testers and Data DevOps, and lead the technical assessment of new talent to ensure a high bar for data engineering excellence.
• Delivery Integration: Work alongside Data Delivery Managers to provide high-confidence technical estimates and risk assessments. You ensure that the data engineering reality aligns with the delivery timeline.

Person specification

• Accomplished expert with extensive hands-on experience at a senior level in building and delivering high-scale data pipelines. You must have a proven track record of implementing automated CI/CD, data quality, and master data management (MDM) processes within a rapid-delivery cloud environment. (Lead Essential Criteria)
• Deep proficiency in the AWS ecosystem and Databricks and strong in writing clean, concise, and parameterized code in Python, SQL, or Scala, with the ability to set the "gold standard" for code maintainability across an organisation.
• Strong expertise in the end-to-end data lifecycle. You have extensive knowledge of data modelling concepts (Data Vault, Star Schema, Medallion) to ensure the physical build of Data Lakes and Lakehouses is performant and scalable.
• Deep familiarity with advanced Databricks features such as Unity Catalogue, Delta Live Tables (DLT), and Spark performance tuning to drive platform efficiency.
• Expert in setting the strategic direction for reusable data engineering frameworks. You have a deep understanding of test automation and containerisation (e.g. Docker, Kubernetes) to ensure that engineering patterns are consistent, repeatable, and decoupled from underlying infrastructure.
• Significant experience in leading and managing engineering teams. You have the interpersonal skills to motivate individuals, and foster a culture of technical excellence.
• Extensive experience mentoring teams in Agile/DevOps delivery. You are comfortable working across multi-disciplinary teams to bridge the gap between technical possibility and delivery reality.
• Demonstrable experience in building or radically maturing data, analytical and test engineering processes from scratch. You know how to implement DevSecOps at scale, moving an organisation from manual deployments to a high-maturity, automated cloud environment.
Desirable
• Hands-on experience working with the Quantexa Decision Intelligence Platform (or similar entity resolution/graph technologies), specifically in optimising network generation and data ingestion patterns.
• Experience managing contracts and relationships with 3rd party software providers and professional services.

Additional information:

A minimum 60% of your working time should be spent at your principal workplace. Although requirements to attend other locations for official business will also count towards this level of attendance.

Behaviours

We'll assess you against these behaviours during the selection process:

• Delivering at Pace

Technical skills

We'll assess you against these technical skills during the selection process:

• Programming and build (Expert)
• Data integration design (Expert)
• Data development process (Expert)
• Communicating between the technical and non-technical (Expert)

Benefits

Alongside your salary of £72,717, Infected Blood Compensation Authority contributes £21,066 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides.

• Premium allowance paid monthly after probation
• Learning and development tailored to your role
• An environment with flexible working options
• A culture encouraging inclusion and diversity
• A Civil Service Pension which provides an attractive pension, benefits for dependants and employer contributions of 28.97%
• 32.5 days of paid annual leave plus 8 bank holidays
• Family friendly policies to support you and your everyday responsibilities
• Enhanced maternity and paternity leave, up to 12 months shared parental leave

Recent changes in skilled worker visa eligibility mean that a Skilled Worker must have a job offer in an eligible skilled occupation from a Home Office-approved sponsor. From 22 July 2025, the job must normally be skilled to level 6 (graduate level) on the Regulated Qualifications Framework for England and Northern Ireland, or the equivalent level in Wales or Scotland, or be included on either the Immigration Salary List or the Temporary Shortage List. Please be aware that if the role is not eligible for a skilled worker visa that we will be unable to provide sponsorship.

Things you need to know

Artificial intelligence

Artificial intelligence can be a useful tool to support your application, however, all examples and statements provided must be truthful, factually accurate and taken directly from your own experience. Where plagiarism has been identified (presenting the ideas and experiences of others, or generated by artificial intelligence, as your own) applications may be withdrawn and internal candidates may be subject to disciplinary action. Please see our candidate guidance (opens in a new window) for more information on appropriate and inappropriate use.

Selection process details

This vacancy is using Success Profiles (opens in a new window), and will assess your Behaviours, Experience and Technical skills.

As part of your application you will be required to provide a CV setting out your career history and qualifications highlighting specific responsibilities and achievements that are relevant for this role.

You will also be required to provide a Statement of Suitability (1000 words max.)

Please use your statement of suitability to explain how you consider your personal skills, qualities and experience provide evidence of your suitability for the role, with reference to the essential criteria in the person specification section of the job advert.

Your CV and Statement of Suitability will be assessed against the essential criteria listed in the 'Person Specification' section of the job advert.

Should a large number of applications be received, an initial sift may be conducted using the statement of suitability. Candidates who pass the initial sift may be progressed to a full sift, or progressed straight to assessment/interview.

Should you be successful at sift, you will be invited to attend an interview where you will be assessed on Behaviours, Technical skills and Experience.

Expected timeline (subject to change)

Expected sift date – WC 30/03/2026
Expected interview date/s – WC 12/04/2026
Interview location - Your interview will either be conducted face to face or by video. You will be notified of the location if you are selected for interview.

Reasonable Adjustment

If a person with disabilities is put at a substantial disadvantage compared to a non-disabled person, we have a duty to make reasonable changes to our processes.

If you need a change to be made so that you can make your application, you should:

• Contact Government Recruitment Service via IBCA.grs@cabinetoffice.gov.uk as soon as possible before the closing date to discuss your needs.
• Complete the ‘Assistance required’ section in the ‘Additional requirements’ page of your application form to tell us what changes or help you might need further on in the recruitment process. For instance, you may need wheelchair access at interview, or if you’re deaf, a Language Service Professional.

Further Information

A reserve list may be held for a period of 12 months from which further appointments can be made.

Any move to IBCA from another employer will mean you can no longer access childcare vouchers. You may however be eligible for other government schemes, including Tax Free Childcare; for further information visit the Childcare Choices website)

Please note that this role requires SC clearance, which would normally need 5 years UK residency in the past 5 years. This is not an absolute requirement, but supplementary checks may be needed where individuals have not lived in the UK for that period. This may mean your security clearance (and therefore your appointment) will take longer or, in some cases, not be possible.

In order to process applications without delay, we will be sending a Criminal Record Check to Disclosure and Barring Service and Disclosure Scotland on your behalf.

However, we recognise in exceptional circumstances some candidates will want to send their completed forms direct. If you will be doing this, please advise Government Recruitment Service of your intention by emailing Pre-EmploymentChecks.grs@cabinetoffice.gov.uk stating the job reference number in the subject heading.

For further information on the Disclosure Scotland confidential checking service telephone: the Disclosure Scotland Helpline on 0870 609 6006 and ask to speak to
the operations manager in confidence, or email Info@disclosurescotland.co.uk

Applicants who are successful at interview will be, as part of pre-employment screening, subject to a check on the Internal Fraud Database (IFD). This check will provide information about employees who have been dismissed for fraud or dishonesty offences. This check also applies to employees who resign or otherwise leave before being dismissed for fraud or dishonesty had their employment continued. Any applicant’s details held on the IFD will be refused employment.

A candidate is not eligible to apply for a role within the IBCA if the application is made within a 5 year period following a dismissal for carrying out internal fraud against government.

If you are experiencing accessibility problems with any attachments on this advert, please contact the email address in the 'Contact point for applicants' section.

PLEASE NOTE: IBCA employees are public servants, therefore the Civil Service Nationality Rules (CSNR) do not apply. Candidates who do not meet the CSNR are welcome to apply and applications will not be rejected: successful candidates however will be subject to a Right to Work check to work within the UK.

Feedback will only be provided if you attend an interview or assessment.

Security

Successful candidates must undergo a criminal record check.Successful candidates must meet the security requirements before they can be appointed. The level of security needed is security check (opens in a new window).

See our vetting charter (opens in a new window).People working with government assets must complete baseline personnel security standard (opens in new window) checks.

Nationality requirements

This job is broadly open to the following groups:

• UK nationals
• nationals of the Republic of Ireland
• nationals of Commonwealth countries who have the right to work in the UK
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities with settled or pre-settled status under the European Union Settlement Scheme (EUSS) (opens in a new window)
• nationals of the EU, Switzerland, Norway, Iceland or Liechtenstein and family members of those nationalities who have made a valid application for settled or pre-settled status under the European Union Settlement Scheme (EUSS)
• individuals with limited leave to remain or indefinite leave to remain who were eligible to apply for EUSS on or before 31 December 2020
• Turkish nationals, and certain family members of Turkish nationals, who have accrued the right to work in the Civil Service

Further information on nationality requirements (opens in a new window)

Working for the Civil Service

Please note this Post is NOT regulated by the Civil Service Commission.The Civil Service embraces diversity and promotes equal opportunities. As such, we run a Disability Confident Scheme (DCS) for candidates with disabilities who meet the minimum selection criteria.

Diversity and Inclusion

The Civil Service is committed to attract, retain and invest in talent wherever it is found. To learn more please see theCivil Service People Plan (opens in a new window) and the Civil Service Diversity and Inclusion Strategy (opens in a new window).

Apply and further information

Once this job has closed, the job advert will no longer be available. You may want to save a copy for your records.

Contact point for applicants

Job contact :

• Name : IBCA Recruitment Team
• Email : ibca.recruitment@ibca.org.uk

Recruitment team

• Email : IBCA.grs@cabinetoffice.gov.uk

Further information

Our recruitment and selection processes are underpinned by the requirement of selection for appointment on the basis of merit by a fair and open competition. If you feel your application has not been treated in accordance with these principles and you wish to make a complaint, you should contact the Resourcing Team at IBCA.grs@cabinetoffice.gov.uk

Attachments

Employee Offer Opens in new window (pdf, 1851kB)