Data Provenance and Lineage in Health Research
Data provenance in health research is the documented origin and history of a dataset: where it came from, who collected it, under what consent and legal basis, and how it has been processed. Data lineage is the traceable path of that data through every transformation, from source system to final result, so any number in a paper or regulatory submission can be followed back to the records and code that produced it. Together they make research reproducible, auditable and defensible to regulators, ethics committees and data custodians.
Why provenance and lineage matter now
Health research increasingly depends on data that was collected for something else. Real-world data from electronic health records, claims, registries and biobanks now supports drug safety studies, external control arms and regulatory decisions. That shift puts the spotlight on data reliability. The FDA’s 2024 guidance on assessing electronic health records and medical claims data for regulatory decision-making asks sponsors to explain how data was collected, curated and transformed. Europe’s DARWIN EU network, coordinated for the European Medicines Agency, depends on standardized and documented data sources across member states.
Clinical trial rules are moving in the same direction. The revised ICH E6(R3) Good Clinical Practice guideline, adopted in January 2025, gives data governance its own emphasis, including audit trails, metadata and traceability across the data life cycle. In Europe, the European Health Data Space (EHDS) Regulation entered into force in 2025 and sets up a framework for secondary use through health data access bodies, which will need to track what data was made available, to whom and for which permitted purpose.
Privacy law adds its own demand. Under the General Data Protection Regulation (GDPR), controllers must be able to demonstrate compliance (the accountability principle in Article 5(2)) and keep records of processing activities under Article 30. Neither is possible if an organization cannot say where personal data came from and what has been done to it.
And the FAIR principles, the most widely adopted framework for research data management, include provenance explicitly. Principle R1.2 states that data and metadata are associated with detailed provenance. Our guide to FAIR data principles in health research covers the full set.
Provenance, lineage and audit trails compared
The three terms are often used interchangeably, but they answer different questions. Compliance teams need all three.
| Concept | Question it answers | Typical content | Who relies on it |
|---|---|---|---|
| Provenance | Where did this data come from and under what terms? | Source institution, collection method, consent basis, legal basis, data sharing agreement, version and date | Data custodians, ethics committees, data protection officers |
| Lineage | How did this result get made? | Transformations, mappings, code versions, parameters, intermediate datasets, dependencies | Researchers, reviewers, regulators, statisticians |
| Audit trail | Who did what, and when? | User actions, access events, approvals, exports, timestamps | Security teams, auditors, data access committees |
| Output record | What left the environment? | Released files, disclosure checks applied, reviewer decisions | Custodians, airlock reviewers, oversight bodies |
How provenance is captured in practice
Standards for describing provenance
The W3C PROV family of specifications, published in 2013, provides a general model for provenance built around three ideas: entities (data), activities (processes that use and generate data) and agents (people, organizations or software responsible). PROV is the basis for many research tools. RO-Crate, a lightweight way to package research data with machine-readable metadata, builds on it, and its Workflow Run profiles record which workflow, inputs and parameters produced a set of outputs.
Workflow managers
Workflow systems widely used in bioinformatics, such as Nextflow, Snakemake and the Common Workflow Language (CWL), can record execution reports and provenance for each run: input files, software containers, parameters and outputs. Running analysis through a workflow manager rather than ad hoc scripts is the single most effective step toward reproducible lineage.
Provenance inside common data models
Harmonization should never erase origin. The OMOP Common Data Model (CDM) keeps provenance in its structure. Source value fields preserve the original code or text from the source system, source concept fields record the original vocabulary concept, and type concept fields record where a fact came from, such as an EHR problem list, a claim or a registry. The CDM_SOURCE table describes the source database and the version of the extract. An OMOP implementation checklist should treat these fields as mandatory rather than optional.
The federated TRE angle
Centralized research platforms create a lineage problem at the very first step. Data is extracted from each source, transferred, transformed during loading, and merged. Every copy and every transfer is a point where provenance can be lost, and the pooled dataset often records only that it came from “site A”, not which extract, under which agreement, with which mapping version.
A federated Trusted Research Environment (TRE) keeps the chain intact. Each custodian holds its own data, harmonizes it locally and records the mapping version used. Approved analysis code travels to the data, so data never leaves the source and there is no extraction step to document. The TRE records which code version ran at which site against which data version, and returns only results.
The last link in the chain is the output. Every result leaving the environment passes through an airlock that checks it for disclosure risk and records what was released, by whom and after which checks. That gives a complete lineage from source record to published figure, with the custodian able to see every step that touched their data.
A practical provenance and lineage framework
- Register every source. For each dataset, record the custodian, collection context, consent model, legal basis, governing agreement, refresh schedule and version identifier.
- Version data and code together. Tag each data release and each analysis code version, and store the pairing for every run.
- Keep source values through harmonization. Preserve original codes and text in OMOP source fields or FHIR extensions, and version your mapping tables.
- Run analyses through workflows. Use a workflow manager and containers so inputs, parameters and software versions are captured automatically.
- Log approvals and access. Link each project to its data access committee decision, ethics approval and the specific datasets approved.
- Record every output. Store what was released, the disclosure checks applied and the reviewer’s decision.
- Make it queryable. Provenance that lives in scattered spreadsheets cannot answer an auditor’s question quickly. Store it in machine-readable form, using PROV or RO-Crate where possible.
Provenance for AI models
Machine learning adds a new kind of derived artifact that needs lineage: the trained model. A clinical prediction model is only as trustworthy as the data it learned from, and reviewers increasingly ask which datasets, which versions, which inclusion criteria and which preprocessing steps produced it. The EU AI Act requires providers of high-risk AI systems to keep technical documentation and to apply data governance practices to training, validation and testing data, which in practice means recording provenance for every dataset a model touched.
Treat a model like any other research output. Record the data versions and code used to train it, the hyperparameters, the evaluation datasets and results, and where the weights are stored. When a model is trained across several sites in a federated TRE, record which sites contributed to each training round and which data version each used. That record lets you answer a question that will come up sooner or later: if a data problem is found at one site, which models are affected?
What this looks like in a real program
National genomics programs illustrate why lineage matters. A variant call in a published study depends on the sample, the sequencing run, the alignment and calling pipeline, the reference genome build and the annotation databases in use at the time. Programs such as Genomics England run analysis inside a TRE where pipelines, versions and outputs are tracked, so a finding can be traced back through each step. The same discipline applies when clinical data is linked in: the mapping version and extract date travel with the result.
For pharma, the stakes show up at submission. When a real-world evidence study informs a regulatory decision, the sponsor must be able to show how the study population was derived from source records. A federated design that records site-level data versions, code versions and airlock decisions produces that evidence as a byproduct of doing the work.
Common pitfalls
Documenting provenance after the fact
Reconstructing lineage months later from memory and file names rarely works. Capture it automatically at the time of processing.
Losing source values during harmonization
Mapping local codes to standard vocabularies without keeping the originals makes mapping errors impossible to trace. Keep both.
Treating consent as a one-time fact
Consent and legal basis can change, for example when a participant withdraws. Provenance records should link to current consent status so that affected data can be identified.
Ignoring outputs
Many organizations track inputs carefully and outputs not at all. The output log is often the record an oversight body asks for first.
What to do next
Start with an inventory of your data sources and check whether each has a complete provenance record: custodian, consent basis, agreement and version. Then pick one active study and try to trace a single reported figure back to source records. The gaps you find are your roadmap. Close them by moving analysis into workflows, preserving source values in your common data model, and routing every output through a logged automated airlock.
Frequently asked questions
What is the difference between data provenance and data lineage?
Provenance describes where data came from and under what terms, including source, consent and legal basis. Lineage describes the path the data took through transformations and analyses to reach a result. Provenance is about origin; lineage is about the journey.
Why is data provenance important in health research?
It makes research reproducible, supports regulatory review of real-world evidence, and lets organizations demonstrate compliance with privacy law and consent terms. Without it, results cannot be traced back or defended.
What standards exist for recording provenance?
The W3C PROV specifications provide a general provenance model. RO-Crate packages research data with machine-readable metadata, including workflow run provenance. The FAIR principles require data to carry detailed provenance under principle R1.2.
How does OMOP preserve data provenance?
The OMOP CDM keeps original codes in source value and source concept fields, records the origin of each fact in type concept fields, and describes the source database and extract version in the CDM_SOURCE table.
How does a federated TRE support data lineage?
Because data stays with each custodian and code travels to it, there is no extraction step to lose track of. The TRE records which code ran against which data version at each site, and the airlock records every output released.
Does GDPR require data provenance?
GDPR does not use the word, but its accountability principle and its requirement to keep records of processing activities mean controllers must know where personal data came from and how it has been processed.