Running LLMs on Sensitive Health Data Inside a TRE
To run a large language model (LLM) on sensitive health data safely, the model must come to the data: host open-weight models inside the Trusted Research Environment (TRE), block all outbound network calls from the workspace, log every prompt and response, and treat model outputs, including generated text and fine-tuned weights, as exports that pass the same airlock checks as any table. Sending patient records to an external model API, however convenient, moves the data outside the environment that was approved to hold it.
Why this matters now
Most clinically useful information sits in free text. Discharge summaries, radiology and pathology reports, clinic letters and nursing notes describe symptoms, reasoning and outcomes that structured codes miss. LLMs are the first tools that can read that text at scale with reasonable accuracy, so research teams want them for phenotyping, cohort building, adverse event detection and extraction of variables that were never coded.
The governance picture has sharpened in parallel. Data use agreements for well-known research datasets now address LLMs directly. PhysioNet, which distributes the MIMIC critical care databases, has published guidance restricting credentialed users from sending its data to third-party online services such as ChatGPT. In the European Union, obligations for providers of general-purpose AI models under the EU AI Act began applying in August 2025, and deployers of AI in health settings face their own requirements. Our summary of the EU AI Act for healthcare research covers what research teams need to do.
There is also a technical reason for care. Research has shown that LLMs can memorize and reproduce parts of their training data. Carlini and colleagues demonstrated in 2021 that verbatim training sequences, including personal information, could be extracted from a large public language model. A model fine-tuned on patient notes can carry that risk out of the environment in its weights.
The Sovereign AI approach: bring the model to the data
Sovereign AI means the data custodian keeps control of both the data and the models that touch it. For LLMs, that rules out the most common pattern, which is calling a commercial API from a notebook. Even with contractual protections, the text leaves the TRE, is processed on infrastructure the custodian does not govern, and may be logged in systems outside any output review.
The alternative is to host the model inside the TRE. Capable open-weight model families are now widely available, and they can run on GPU nodes within the same secure boundary as the data. The model weights are imported through a controlled ingress process, scanned and approved like any other software, and then used with no network route out. The data never leaves the source, and neither do the prompts that contain it.
In a federated TRE, this extends across institutions. The same approved model and extraction prompt can be deployed at every participating hospital or biobank. Each site runs extraction locally on its own notes, converts the results into structured variables, and returns only aggregate, checked outputs. Our guide to the agentic federated TRE explains how AI agents operate under the same controls.
Deployment patterns compared
| Pattern | Where the data goes | Output control | Suitability for sensitive health data |
|---|---|---|---|
| Public consumer chatbot | Outside the organization, to a consumer service | None | Not appropriate |
| Commercial API with enterprise terms | To the vendor’s cloud under contract | Contractual only, outside the TRE airlock | Rarely acceptable for identifiable data; depends on approvals and law |
| Private model endpoint in the organization’s cloud | Stays in the organization’s tenancy, but often outside the TRE boundary | Partial, depends on network design | Possible with strict network isolation and logging |
| Open-weight model hosted inside the TRE | Stays inside the approved research environment | Full, every output passes the airlock | Strong fit for single-custodian research |
| Open-weight model deployed across a federated TRE | Stays at each custodian | Full, per-site airlock plus aggregate review | Strongest fit for multi-site and cross-border research |
A practical guide to running LLMs inside a TRE
1. Define the task and the output first
Most research uses of LLMs are extraction or classification: reading a note and producing a structured answer, such as “smoking status: former” or “metastatic: yes”. Define the target variables and their allowed values before choosing a model. Structured outputs are far easier to validate and to check for disclosure than free text.
2. Choose and import the model
Select an open-weight model whose license permits your use, and whose size fits the GPUs available inside the environment. Smaller models fine-tuned or carefully prompted for a narrow extraction task often match much larger general models. Import weights through the TRE’s ingress review and record the exact version and checksum.
3. Isolate the runtime
Run inference servers inside the workspace network with no outbound internet access. Disable telemetry in any serving framework. Researchers should reach the model only through an internal endpoint that the TRE controls.
4. Log prompts and responses
Keep an audit log of every prompt, response, model version and user inside the environment. This supports reproducibility, error analysis and investigation if something goes wrong. The logs contain sensitive text, so they stay inside the TRE.
5. Validate against human annotation
Annotate a sample of documents by hand and measure precision and recall for each variable. Repeat at each site in a federation, since documentation style differs between hospitals and a prompt that works in one may fail in another.
6. Govern fine-tuning carefully
Fine-tuning on patient notes improves accuracy but raises the memorization risk. Keep fine-tuned weights inside the TRE by default. If weights must leave, assess them for leakage and consider training with differential privacy, accepting some loss of accuracy.
7. Put every output through the airlock
Generated text can quote a patient’s note verbatim, including names, dates or rare conditions. Aggregate tables derived from LLM extraction can still contain small cells. Output checking and statistical disclosure control apply to LLM results exactly as they apply to any other analysis.
8. Plan for model updates
New model versions appear often, and a better model can change extracted values. Treat a model change like any other change to an analysis pipeline: re-run validation, record the new version, and note which study results were produced with which model. Reproducibility depends on being able to say exactly which weights and prompts produced a given dataset.
What this looks like in practice
Consider a pharma research team studying treatment patterns in a cancer where key facts, such as biomarker status and reasons for stopping therapy, are recorded only in clinic letters. In a centralized model, the team would need notes from several hospitals copied into one place, then sent to an LLM service. In a federated TRE, each hospital keeps its notes. The team submits an approved extraction pipeline built around a hosted open-weight model. It runs at each site, converts notes into OMOP-aligned variables, and the team receives aggregate counts and survival estimates that have passed each site’s airlock. The notes, the prompts and the model’s raw responses never leave the hospitals.
This is the same federated architecture that national programs and life sciences organizations such as Genomics England and Boehringer Ingelheim rely on for sensitive data analysis with Lifebit. LLMs become one more tool inside the environment rather than a reason to take data out of it.
Common pitfalls
“De-identified notes are safe to send out”
Automated de-identification of free text is imperfect. Names, addresses and rare details slip through, and combinations of clinical facts can identify a person. Treat de-identified notes as sensitive unless formally assessed otherwise.
Retrieval systems that bypass controls
Retrieval-augmented generation (RAG), where the model searches a document index before answering, is useful inside a TRE. But the index is a copy of sensitive text. Keep it inside the environment and apply the same access rules as the source data.
Trusting fluent output
LLMs produce confident, well-formed answers that can be wrong. Validation against annotated samples is required, not optional, before extracted variables feed any analysis.
Ignoring agent tool access
If an LLM agent can call tools, such as running queries or writing files, its permissions must be scoped like a researcher’s. An agent with broad access is an output risk waiting to happen.
Prompt injection from source text
Clinical documents, scanned letters and patient-submitted forms can contain text that an LLM reads as an instruction. Inside a TRE with no network egress the damage is limited, but an injected instruction could still distort extraction or push sensitive text into an output. Constrain the model to structured output formats and validate results rather than trusting free-form answers.
What to do next
Start with one well-defined extraction task on one dataset. Stand up a hosted open-weight model inside your TRE with no outbound access, build an annotated validation set, and route all results through your existing airlock. Document the model version, prompts and validation results so the approach can be reviewed by your data access committee. Once it works at one site, the same package can be deployed across a federation. Our piece on why your AI should stay where the data lives sets out the broader case.
Frequently asked questions
Can I use ChatGPT or another public LLM on patient data?
Not on identifiable or sensitive patient data in a research setting unless your approvals, contracts and applicable law explicitly allow it. Many data use agreements prohibit it. The safer approach is to host an open-weight model inside the Trusted Research Environment that already holds the data.
What is an open-weight model?
An open-weight model is a language model whose trained parameters are published so that organizations can download and run it on their own infrastructure. This makes it possible to run the model inside a secure environment with no external network access.
Do LLM outputs need airlock review?
Yes. Generated text can repeat identifying details from source notes, and tables built from LLM extraction can contain small counts. Every output leaving a TRE should be checked, whether it came from a statistical model or a language model.
Is fine-tuning an LLM on health records risky?
It can be. Language models can memorize training data, so fine-tuned weights may contain information about individuals. Keep fine-tuned models inside the TRE by default and assess leakage risk before any release.
How do LLMs work in a federated TRE?
The same approved model and pipeline are deployed at each participating site. Each site runs the model locally on its own data, and only aggregate, checked results are returned. No site shares its notes with any other party.
What hardware is needed to run an LLM inside a TRE?
Inference usually requires GPU nodes inside the secure environment. Model size determines the memory needed, and smaller models tuned for a narrow task often perform well enough while keeping infrastructure costs manageable.