External Validation of Clinical AI Models Across Sites
External validation of a clinical AI model means testing it on data from sites, populations, or time periods that played no part in its development, and reporting discrimination, calibration, and clinical utility separately for each. It is the only reliable way to find out whether a model that performed well where it was built will hold up in a new hospital. The most practical way to do it across many sites is to run the frozen model inside each site’s own environment and return only performance metrics, so no patient data has to move.
Why this matters now
Clinical prediction models often lose accuracy when they move. Differences in patient mix, measurement practices, coding habits, equipment, and care pathways all shift the data a model sees. A widely cited example is the external validation of a proprietary sepsis prediction model embedded in a major electronic health record (EHR) vendor’s product: Wong and colleagues, writing in JAMA Internal Medicine in 2021, reported an area under the curve of 0.63 at their academic health system, substantially lower than the developer’s reported performance, and found the model missed many sepsis cases while generating frequent alerts.
Reporting standards and regulation now expect external evidence. TRIPOD+AI, published in The BMJ in 2024, updates the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement for machine learning and asks authors to describe validation data and performance in detail. PROBAST, the Prediction model Risk Of Bias ASsessment Tool, flags models validated only on development data as high risk of bias. The EU AI Act classifies many clinical AI systems as high risk and requires evidence of accuracy and robustness across their intended conditions of use. The US Food and Drug Administration (FDA) finalized guidance on predetermined change control plans for AI-enabled device software functions in 2024, which assumes manufacturers can monitor and revalidate models after deployment.
All of this requires access to data the developer does not own. That is where most validation efforts stall.
The Sovereign AI angle: validate where the data lives
A hospital asked to validate an external model faces two bad options in a centralized world. It can send an extract of its patients to the model developer, which raises governance, contractual, and privacy objections. Or it can accept the developer’s published metrics, which may not reflect its own population.
A federated Trusted Research Environment (TRE) offers a third option. The model, packaged as a frozen container with its preprocessing code, is sent to each participating site. It runs against locally harmonized data, and only aggregate performance statistics pass back through an automated airlock. The data never leaves the source, and the hospital keeps full control over what is released. This is Sovereign AI in the evaluation stage: each custodian decides whether a model is fit for its patients using its own data, under its own governance.
The same pattern supports repeated validation over time. Once a model is deployed, the site can rerun the evaluation container monthly or quarterly to detect drift without any new data sharing agreement. For more on how federated evaluation relates to federated training, see federated analytics vs federated learning.
A practical framework for multi-site external validation
1. Define the validation question
Distinguish the type of validity you are testing. Geographic validation tests a new site. Temporal validation tests a later period at the same site. Domain validation tests a different setting, for example moving from intensive care to general wards. Each answers a different question and should be reported separately.
2. Freeze and package the model
Validation is only meaningful if the model does not change. Package the trained model, its preprocessing pipeline, feature definitions, and version identifier in a container. Any site-level recalibration is a separate, documented step after the untouched model has been evaluated.
3. Harmonize inputs
Most apparent performance drops at new sites come from input mismatch rather than model failure: a lab test in different units, a diagnosis coded with a different vocabulary, a missing feature silently imputed. Mapping each site to a common data model such as the Observational Medical Outcomes Partnership (OMOP) Common Data Model, and publishing the exact concept sets the model uses, makes inputs comparable. Report feature missingness per site alongside performance.
4. Compute the right metrics locally
Each site computes a fixed set of metrics inside its environment:
- Discrimination, such as the area under the receiver operating characteristic curve (AUROC) and, for rare outcomes, the area under the precision-recall curve.
- Calibration, including calibration slope, calibration-in-the-large, and a binned or smoothed calibration curve. Van Calster and colleagues described calibration as the Achilles heel of predictive analytics in 2019; a model can rank patients well and still overestimate risk badly.
- Clinical utility, through decision curve analysis at clinically relevant thresholds.
- Subgroup performance by age band, sex, ethnicity where recorded, and other relevant strata, with small cells suppressed.
5. Release and combine through output checks
Aggregate metrics and calibration curves pass an airlock that applies statistical disclosure control, for example suppressing subgroups below a minimum count. Our guide to output checking and statistical disclosure control explains the typical rules. A coordinating team then pools site estimates with random-effects meta-analysis, which reports both an average performance and the between-site heterogeneity. That heterogeneity figure is often more informative than the average.
6. Decide and document
Each site decides whether to adopt, recalibrate, or reject the model for local use, based on its own results. Record the decision, the evidence, and the model version.
7. Plan for revalidation
External validation is not a one-time gate. Case mix changes, coding systems are updated, new assays replace old ones, and clinical practice responds to the model itself once it is in use, which can change the outcome rates it was trained to predict. Set a revalidation schedule before deployment, name the metrics that will trigger review, and keep the evaluation container versioned so the same code runs each time. For teams working under European rules, our summary of the EU AI Act and healthcare AI sets out the documentation and post-market monitoring expectations that this schedule supports.
Validation approaches compared
| Approach | What it tests | Data movement | Main limitation |
|---|---|---|---|
| Internal validation (cross-validation, bootstrap) | Overfitting within the development data | None beyond development | Says little about new sites |
| Centralized external validation | Performance on pooled external data | Extracts sent to developer or evaluator | Slow agreements, custodians often decline |
| Site-run validation, ad hoc | Performance at one new site | None | Inconsistent metrics and code across sites |
| Federated external validation in a TRE | Performance at every participating site with identical code | Only checked aggregate metrics | Requires harmonized data and container infrastructure at each site |
| Continuous federated monitoring | Drift over time after deployment | Only checked aggregate metrics | Needs outcome data to accrue before metrics are stable |
Real-world context
Large national data programs increasingly provide the infrastructure external validation needs. Genomics England and CanPath, both Lifebit customers, give approved researchers governed access to clinical and genomic data within trusted environments instead of releasing copies, which is the property a validation study needs: the evaluator can run code against real populations while the custodian stays in control. Research consortia built on the OMOP Common Data Model, notably the Observational Health Data Sciences and Informatics (OHDSI) network, have repeatedly shown that the same analysis package can run across dozens of databases in different countries and return comparable results. External validation of prediction models is one of the use cases OHDSI has published on, using its patient-level prediction framework.
In industry, Boehringer Ingelheim works with Lifebit on reuse of clinical data in early discovery. Any organization developing models on its own data faces the same question before a model is used elsewhere: will it still work on someone else’s patients?
Common pitfalls and objections
Reporting only discrimination
AUROC is necessary but not sufficient. A model with a good AUROC and poor calibration will produce misleading absolute risks, which matters whenever a threshold drives action.
Quietly recalibrating before evaluating
Refitting the intercept or thresholds at a new site and then reporting performance hides the transportability problem. Report the untouched model first, then any recalibrated version, clearly labeled.
Ignoring outcome definition differences
If sepsis, readmission, or deterioration is defined differently at the validation site, the model is being judged against a different target. Agree outcome phenotypes in advance and run them as part of the container.
Too few events
Validation studies need enough outcome events for stable estimates; methodological guidance suggests a minimum of around 100 events, and preferably more, for reliable calibration assessment. Federating across sites is often the only way to reach that number for rare outcomes.
What to do next
Pick one model that is already in use or close to deployment, and list the sites whose patients differ most from the development population. Agree the metrics above, the outcome definition, and the disclosure rules, then run a frozen container at two or three of those sites. The results will show whether your harmonization and packaging are good enough to scale. If you are assessing infrastructure for this, our overview of the federated Trusted Research Environment explains how code execution and output review work at each site.
Frequently asked questions
What is the difference between internal and external validation?
Internal validation estimates performance using the development data, for example by cross-validation or bootstrapping. External validation tests the model on data from different sites, populations, or time periods that were not used in development, which is a much stronger test of whether it generalizes.
Which metrics should an external validation report?
At minimum, discrimination (such as AUROC), calibration (calibration slope, calibration-in-the-large, and a calibration curve), and clinical utility through decision curve analysis, reported overall and for key subgroups at each site.
How many sites are needed for external validation?
There is no fixed number, but several heterogeneous sites give a far better picture than one. Random-effects meta-analysis across sites estimates both average performance and how much it varies, which is what a new adopter needs to know.
Can a model be validated at a hospital without the hospital sharing data?
Yes. In a federated Trusted Research Environment the frozen model runs inside the hospital’s environment and only aggregate metrics are released after output checks. Patient records remain with the hospital.
What should happen if a model performs poorly at a new site?
First check for input problems such as unit mismatches or missing features. If the model is correctly applied and still underperforms, the site can recalibrate it, retrain it with local data, or decline to use it. Each option should be documented.
Does the EU AI Act require external validation?
The Act requires high-risk AI systems to achieve appropriate accuracy and robustness and to document testing. It does not prescribe a specific validation design, but external validation across representative settings is the most direct way to provide that evidence for clinical models.