Training Foundation Models on Health Data Without Moving It
You can train or adapt a foundation model on health data without centralizing it by sending the model to each data custodian, training locally inside a federated Trusted Research Environment (TRE), and aggregating only model updates that have passed privacy controls. In practice most health organizations start from an openly available pretrained model and fine-tune it across sites, because full pretraining from scratch is rarely necessary and far harder to govern. The patient records stay where they were collected, and only reviewed weights, gradients, or evaluation metrics cross the boundary.
Why this matters now
A foundation model is a large model trained on broad data that can be adapted to many downstream tasks. The term was popularized by the Stanford Center for Research on Foundation Models in 2021, and it now covers large language models, vision transformers for radiology and pathology, and models trained on genomic sequences or longitudinal electronic health records (EHRs). Every one of these model families improves with more diverse data, and in health that diversity sits in hospitals, biobanks, and national registries that cannot legally or ethically pool their records.
Three pressures have arrived at the same time. First, the EU AI Act places obligations on providers of general-purpose AI models, including documentation of training data, and classifies many clinical uses as high risk. Second, the May 2026 UK Biobank incident showed what happens when approved researchers can export data through a centralized platform’s normal workflow: nobody broke a rule, yet the data left. Third, peer-reviewed work has shown that large models can memorize and regurgitate fragments of their training data (Carlini and colleagues demonstrated training-data extraction from a production language model in 2021). A copy of a health dataset sitting in a model developer’s cloud bucket is now two risks: the copy itself and whatever the trained weights retain.
The Sovereign AI angle
Sovereign AI means a country, health system, or institution keeps legal and operational control over the data, the compute, and the resulting models. For foundation models, that control has to cover more than storage location. It has to cover who can start a training run, what code runs, what leaves the environment, and who owns the adapted weights.
The federated pattern delivers that control because the data never leaves the source. Compute is dispatched to each custodian’s environment, runs against locally harmonized data, and returns only outputs that an automated airlock has checked. Lifebit holds US Patent 12,519,781 on this federation approach, and the same architecture that supports federated statistics also supports federated model training. If you want the basics of how training differs from querying, our explainer on federated analytics vs federated learning covers the distinction.
Why fine-tuning is the realistic target
Pretraining a large model from scratch needs enormous volumes of tokens or images and sustained accelerator time. Health custodians rarely have either at a single site, and synchronizing full pretraining across sites is slow and expensive. Fine-tuning a pretrained model is different. Parameter-efficient methods such as low-rank adaptation (LoRA), introduced by Hu and colleagues in 2021, train a small set of additional weights while the base model stays frozen. The update each site sends back is small, which makes it cheaper to transmit and easier to inspect.
Why the model is also sensitive
Once a model has been adapted on patient data, its weights are a derived data product. A Sovereign AI program should treat them that way: stored inside the governed environment, released only through review, and logged in the same audit trail as any other output.
A practical framework for federated foundation model training
The steps below assume a network of custodians, each running a federated TRE node, and a coordinating team that owns the training plan.
- Choose the base model and license. Select an openly licensed pretrained model whose license permits fine-tuning and redistribution of derived weights. Record the base model version, because every later evaluation depends on it.
- Harmonize locally. Each site maps its data to a common model, typically the Observational Medical Outcomes Partnership (OMOP) Common Data Model for structured EHR data, and Fast Healthcare Interoperability Resources (FHIR) for exchange. Without harmonization, the model learns site coding habits instead of clinical signal.
- Agree the training protocol. Define the task, the objective, the number of federated rounds, the aggregation rule (federated averaging, described by McMahan and colleagues in 2017, is the usual starting point), and the privacy budget if you apply differential privacy.
- Containerize and approve the code. Training code runs as a reviewed container inside each node. The data access committee approves the code, not just the project description.
- Train locally, aggregate centrally. Each node computes an update against its own data. Updates pass the airlock and are combined by an aggregator that never sees raw records.
- Evaluate at every site. Hold out local test sets and report performance per site, not only pooled. A model that works on average can fail badly at one hospital.
- Release through review. Final weights and model cards leave only after disclosure checks and sign-off from each custodian.
Centralized vs federated training compared
| Dimension | Centralized training | Federated training in a TRE |
|---|---|---|
| Where patient data sits | Copied to the model developer’s environment | Stays with each custodian |
| What crosses the boundary | Full records or extracts | Model updates and metrics after airlock review |
| Legal agreements | Data transfer agreements per site, often slow to negotiate | Code and output approval under each custodian’s existing governance |
| Memorization exposure | Weights trained on the full pooled copy | Reduced by local training, clipped updates, and optional differential privacy |
| Heterogeneity handling | Hidden inside the pooled dataset | Visible per site, which supports fairness and drift analysis |
| Engineering effort | Simpler pipeline, one cluster | Needs orchestration, harmonization, and secure aggregation |
| Sovereignty | Control shifts to whoever holds the copy | Each custodian keeps control of data and derived weights |
Real-world context
National genomics and biobank programs have already shown that large-scale research can run on data that stays in a governed environment. Genomics England, a Lifebit customer, provides researcher access to whole-genome and clinical data through a TRE rather than by releasing copies, and CanPath in Canada, also a Lifebit customer, uses federated infrastructure to give researchers access across provincial cohorts. These are the same foundations a federated foundation-model program needs: harmonized data, approved code execution, and output review.
In industry, Boehringer Ingelheim works with Lifebit on reuse of clinical data for early discovery, a setting where training and evaluating models against internal trial data without creating uncontrolled copies is a practical requirement. Public research programs point the same way. The All of Us Researcher Workbench and the UK Biobank Research Analysis Platform both moved researchers toward cloud workspaces instead of bulk downloads, and the May 2026 incident showed that workspace access alone is not enough if the platform still lets data walk out. Automated output checking is the missing control.
For a deeper operational view of running these jobs in a pharma setting, see our federated AI model training guide for biopharma.
Common pitfalls and objections
“Federated training is too slow for large models”
It is slow if you try to federate full pretraining. It is workable when you federate fine-tuning of a small adapter, because each round moves megabytes rather than the full model. Plan for fewer, larger local training steps per round.
“Model updates can still leak data”
They can. Gradient inversion attacks are documented in the literature. The mitigations are well understood: clip and add noise to updates (differentially private stochastic gradient descent, described by Abadi and colleagues in 2016), use secure aggregation so the coordinator sees only the combined update, and run membership-inference tests on the final model before release. Treat these as defaults, not options.
“Our sites use different coding systems”
This is the most common real failure. Non-harmonized inputs produce a model that performs well at the largest site and poorly elsewhere. Invest in OMOP mapping and vocabulary alignment before the first training round, and measure mapping coverage per site. Free-text data needs the same care: discharge summaries and radiology reports differ in template, abbreviation, and language between hospitals, so agree tokenization and de-identification rules centrally and apply them identically at every node. A mapping report per site, reviewed before training starts, costs a few days and saves weeks of debugging a model that quietly learned one hospital’s shorthand.
“We cannot inspect what the model learned”
Per-site evaluation, subgroup metrics, and a model card that documents data sources and limits are the practical answer. They also map directly onto EU AI Act documentation expectations; our note on the EU AI Act and healthcare AI sets out what research teams must record.
What to do next
Start with an inventory rather than a model. List the custodians you need, the data each holds, whether it is already mapped to OMOP, and what each governance body would need to approve code execution and weight release. Then pick one narrow, measurable task, such as adapting a clinical language model to classify discharge summaries, and run it across two or three sites with full airlock review. The lessons from that pilot, especially on harmonization gaps and update review, will shape the larger program more than any architecture diagram.
If you are assessing platforms for this work, our overview of the federated Trusted Research Environment explains how data access, code approval, and output checking fit together in one governed system.
Frequently asked questions
Can you pretrain a foundation model from scratch on federated health data?
It is technically possible but rarely practical. Full pretraining needs very large data volumes and long training runs, and synchronizing that across sites is slow. Most programs fine-tune an openly licensed pretrained model across sites, which delivers most of the benefit at a fraction of the cost and governance burden.
What leaves each site during federated fine-tuning?
Only model updates, such as adapter weights or gradients, and aggregate evaluation metrics. These pass an automated airlock that checks them against disclosure rules before they reach the aggregator. Patient records and extracts do not leave.
Does federated training eliminate privacy risk?
No. It removes the risk of a central data copy, but updates and final weights can still carry information. Differential privacy, secure aggregation, update clipping, and membership-inference testing reduce that residual risk to a level custodians can govern.
Why does data harmonization matter for foundation models?
If each site codes diagnoses, labs, and medications differently, the model learns site-specific conventions rather than clinical patterns. Mapping to a common model such as OMOP before training makes updates comparable and improves performance across sites.
Who owns the fine-tuned model weights?
That is set by the collaboration agreement, but in a Sovereign AI setup the weights are treated as a derived data product. They stay inside the governed environment until each custodian approves release, and every release is logged.
How is this different from a centralized cloud workspace?
A centralized workspace still holds a copy of the data under one operator, and users can often export results with limited checks. A federated TRE keeps data at each custodian and routes every output through review, which closes the export path that caused the May 2026 UK Biobank incident.