Genomic Variant Harmonization: VCF Normalization & Liftover

Harmonizing genomic variant data means making variant calls from different sources directly comparable, which comes down to two steps: normalizing every Variant Call Format (VCF) file so the same variant is always written the same way, and placing every dataset on the same reference genome, usually by lifting GRCh37 coordinates over to GRCh38. Normalization splits multiallelic records, left-aligns indels, and trims redundant bases against the reference. Liftover converts positions between assemblies using chain files and then fixes the reference allele and strand issues that coordinate conversion alone leaves behind.

Why variant harmonization matters now

Genomic studies increasingly combine cohorts that were sequenced years apart, on different platforms, and processed with different pipelines. Older datasets were often aligned to GRCh37 (also known as hg19), while most current national programs and major reference resources use GRCh38, which the Genome Reference Consortium released in December 2013. Population frequency resources show the shift: recent gnomAD releases are built on GRCh38. A variant that looks novel in one cohort may simply be written differently, or sit on a different build, from the same variant in another.

The cost of getting this wrong is quiet. Mismatched representations do not usually cause errors; they cause missed matches. An association signal splits across two records, a pathogenic indel fails to match its ClinVar entry, or allele frequencies look lower than they really are. In a federated setting, where analysts cannot inspect raw files at every site, those silent mismatches are especially hard to catch after the fact. Harmonization has to be done correctly at the source.

Step one: VCF normalization

Why the same variant has many spellings

VCF lets the same genomic change be represented in more than one way. An insertion in a repetitive region can be placed at several equivalent positions. A record can carry extra flanking bases that are not part of the change. Several alternative alleles can be packed into one line. Tan, Abecasis, and Kang described this problem in a 2015 Bioinformatics paper and defined a normalized form: a variant is normalized when it is parsimonious (represented with the fewest bases possible) and left-aligned (shifted as far left as it can go while describing the same sequence).

The normalization steps

1. Split multiallelic records. Break records with several alternative alleles into one record per allele, adjusting genotype and per-allele fields such as AD and AF. With bcftools this is bcftools norm -m -any.
2. Left-align and trim against the reference. Supply the exact reference FASTA used for calling, for example bcftools norm -f reference.fa. The tool shifts indels left and removes shared padding bases.
3. Check reference alleles. Confirm that each REF allele matches the reference sequence at its position. Mismatches point to a build mix-up or a corrupted file. The --check-ref option in bcftools can warn, exclude, or fix them.
4. Decompose complex variants if your analysis needs it. Multi-nucleotide polymorphisms and block substitutions can be split into single-base changes with tools such as vt. Whether to do this depends on downstream use, so record the choice.
5. Remove exact duplicates. After splitting and aligning, previously distinct records may become identical. Deduplicate them.

Two practical details cause many problems. First, contig naming: some files use “chr1” and others use “1”. Pick one convention and rename consistently. Second, the reference FASTA must match the calling reference exactly, including decoy and alternate contigs. Normalizing against the wrong FASTA produces confident nonsense.

For interchange with other systems, the GA4GH Variation Representation Specification (VRS) uses a fully justified normalization, which represents ambiguous indels by the full span of equivalent positions rather than a single left-aligned one. Know which convention each downstream resource expects.

Step two: GRCh37 to GRCh38 liftover

How liftover works

Liftover maps coordinates from one assembly to another using a chain file, which describes aligned blocks between the two builds. The UCSC liftOver tool and CrossMap both use chain files. For VCFs, Picard LiftoverVcf is widely used because it also handles allele-level problems: when a region maps to the opposite strand, it reverse-complements the alleles, and when the reference base differs between builds, it can swap REF and ALT and update genotypes accordingly.

What liftover cannot fix

Some GRCh37 positions have no counterpart in GRCh38, or map to more than one place. GRCh38 corrected errors in GRCh37, added sequence, and introduced many alternate haplotype contigs. Variants in affected regions are dropped or flagged during liftover. Expect a small fraction of records to fail and review what fails, because losses cluster in regions that can matter clinically.

After liftover, normalize again. Coordinates change, the surrounding sequence can differ, and an indel that was left-aligned on GRCh37 may not be left-aligned on GRCh38.

Liftover or realignment?

If raw reads (FASTQ, BAM, or CRAM) are available and compute budget allows, realigning to GRCh38 and recalling variants gives better results than liftover, especially in regions that changed between builds. Liftover is the right tool when only VCFs exist, when turnaround matters, or when the goal is matching against a resource rather than fresh discovery.

A harmonization framework: choosing the right approach

That last row matters. When single-sample VCFs are merged, a site absent from one sample could mean homozygous reference or no coverage. Genomic VCFs (gVCFs) carry reference blocks that resolve the difference. Merging plain VCFs without that information inflates apparent heterogeneity between cohorts.

The federated harmonization angle

Most harmonization guides assume you can gather every VCF in one place and run a single pipeline. In many national and clinical programs you cannot. Genomic data is among the most sensitive data there is, and custodians increasingly keep it within their own infrastructure. In a federated Trusted Research Environment (TRE), the harmonization pipeline goes to each site instead: the same containerized workflow, the same tool versions, and the same reference FASTA and chain file run inside each custodian’s environment. The data never leaves the source, yet every site produces variants in an identical representation on an identical build.

That approach needs discipline. Pin every tool and reference by version and checksum. Emit a manifest from each site describing the pipeline run, the number of records in and out at each step, and the liftover failure rate. Compare those manifests centrally; a site with an unusual failure rate or REF mismatch count usually has a reference or build problem. Lifebit’s federated TRE runs versioned workflows at each custodian and releases only checked aggregate outputs through its automated airlock, which gives analysts that quality signal without exposing individual genotypes. Our guide on how to analyze genomic data without moving it covers the workflow pattern in more depth.

Variant harmonization also has to meet phenotype harmonization. Genotypes become useful when they can be joined to consistent clinical data, which is why Lifebit pairs genomic pipelines with automated mapping of clinical records to standards such as the OMOP Common Data Model. For the broader picture, see what data harmonization means.

Real-world context

Large national programs have shown both the value and the difficulty of reference transitions. Resources such as gnomAD, ClinVar, and dbSNP provide GRCh38 coordinates, and many clinical annotation tools now treat GRCh38 as the default. Lifebit works with genomic custodians including Genomics England, where population-scale sequencing and strict governance make federated, reproducible pipelines a practical requirement rather than an option. For the operations side of that work, see our article on federated genomic analysis at population scale.

Common pitfalls

Normalizing once, not twice

Teams normalize before liftover and forget to do it after. Coordinates and flanking sequence change between builds, so a second pass is necessary.

Ignoring failed records

Silently dropping variants that fail liftover hides losses in clinically relevant regions. Keep the rejected records, count them, and report them.

Mixing references

Using a GRCh38 FASTA with or without alternate contigs, or with a different decoy set, from the one used for calling produces REF mismatches that look like data errors.

Forgetting strand in array data

A/T and C/G SNPs cannot be resolved by allele alone. Use allele frequency comparison with a reference panel, or drop them if frequencies are close to 50 percent.

What to do next

Audit what you have. For each cohort, record the build, the reference FASTA, the calling pipeline and version, whether raw reads exist, and whether files are VCF or gVCF. That inventory tells you which row of the framework table applies to each dataset. Then define one harmonization pipeline, containerize it, and run it everywhere, including at sites whose data you will never see directly. For the architecture that makes cross-site pipelines practical, read Lifebit’s overview of the federated Trusted Research Environment.

Frequently asked questions

What does it mean to normalize a VCF?

Normalizing a VCF means rewriting variants into a single canonical form: splitting multiallelic records into one allele per line, left-aligning indels, and trimming redundant bases against the reference genome. It ensures the same variant is always represented the same way.

Should I normalize before or after liftover?

Both. Normalize before liftover so records are clean and consistent, then normalize again afterwards because coordinates and flanking sequence change between GRCh37 and GRCh38.

Which tool is best for lifting over VCFs?

UCSC liftOver and CrossMap handle coordinate conversion with chain files. Picard LiftoverVcf is often preferred for VCFs because it also reverse-complements alleles on strand flips and can swap REF and ALT when the reference base differs between builds.

Is realignment better than liftover?

When raw reads are available, realigning to GRCh38 and recalling variants is generally more accurate, particularly in regions that changed between builds. Liftover is appropriate when only VCFs exist or when speed matters more than maximum accuracy.

Why do some variants fail liftover?

Some GRCh37 regions have no equivalent in GRCh38, map to multiple locations, or fall in sequence that was corrected or restructured. Those records are rejected or flagged and should be reviewed rather than silently discarded.

Can variant harmonization be done without centralizing data?

Yes. In a federated TRE, the same versioned pipeline, reference, and chain file run at each custodian’s site. Each site produces harmonized variants locally, and only quality metrics and approved aggregate results are shared.