Running a GATK Germline Variant Calling Pipeline on Your Own Genome
By the end of this guide you will have a filtered, normalized and annotated VCF for one human genome. You will also have the quality metrics you need to decide whether to trust it. You need paired-end FASTQ files from a 30x or deeper whole-genome run, a machine with at least 16 cores and 64 GB of RAM, and roughly 500 GB of free disk for intermediates. You also need the GATK resource bundle for GRCh38 and GATK 4.5 or later. The tools bwa-mem2, samtools and bcftools are required as well. Everything below assumes a single sample, which changes several decisions compared to the population-scale workflows the Broad documents.
1. Set up the reference and resource bundle
The reference you choose determines which coordinates every downstream tool speaks, so pick one and stay with it. We use GRCh38_full_analysis_set_plus_decoy_hla.fa, the same analysis set used by the 1000 Genomes Project and the Broad’s production workflows. It includes decoy contigs and HLA contigs that absorb reads which would otherwise pile up as false positives in the main chromosomes.
Index it once:
bwa-mem2 index GRCh38_full_analysis_set_plus_decoy_hla.fa
samtools faidx GRCh38_full_analysis_set_plus_decoy_hla.fa
gatk CreateSequenceDictionary -R GRCh38_full_analysis_set_plus_decoy_hla.fa
The bwa-mem2 index files for a human reference total roughly 30 GB on disk, and the aligner memory-maps them. Budget about 40 GB of RAM for the alignment step alone. From the GATK resource bundle you need four files. These are Homo_sapiens_assembly38.dbsnp138.vcf, Mills_and_1000G_gold_standard.indels.hg38.vcf.gz, Homo_sapiens_assembly38.known_indels.vcf.gz and wgs_calling_regions.hg38.interval_list.
2. Align with bwa-mem2
Alignment is where most irrecoverable errors happen, because a read placed in the wrong repeat copy produces a variant call that looks perfectly well supported. Two flags matter more than the rest. -K 100000000 fixes the batch size so output is deterministic regardless of thread count, and -Y uses soft clipping for supplementary alignments so the full read sequence is retained.
bwa-mem2 mem -t 32 -K 100000000 -Y \
-R '@RG\tID:S1.L1\tSM:S1\tLB:lib1\tPL:ILLUMINA\tPU:HXXXXDSX.1.ATCACG' \
GRCh38_full_analysis_set_plus_decoy_hla.fa \
S1_L1_R1.fastq.gz S1_L1_R2.fastq.gz \
| samtools sort -@ 8 -m 3G -o S1.L1.bam -
Write one read group per lane and per library. GATK uses the read group to model error rates separately for each sequencing run, and a single merged read group across four lanes will blur real differences between them. The PU field should encode flowcell, lane, and barcode. On 32 cores this takes roughly two to four hours for a 30x genome.
If you have multiple lanes, align each separately and merge with samtools merge before duplicate marking.
3. Mark duplicates
PCR and optical duplicates inflate apparent read support, which is exactly the evidence the caller uses to decide between a sequencing error and a real allele. Marking them lets the caller count independent observations rather than copies.
gatk MarkDuplicates \
-I S1.merged.bam \
-O S1.dedup.bam \
-M S1.dup_metrics.txt \
--OPTICAL_DUPLICATE_PIXEL_DISTANCE 2500 \
--CREATE_INDEX true
The pixel distance of 2500 is the value to use for patterned flowcells such as NovaSeq and HiSeq X. The default of 100 is for older unpatterned flowcells and will badly undercount optical duplicates on modern instruments. Read S1.dup_metrics.txt and check PERCENT_DUPLICATION. For a PCR-free 30x library, anything above about 0.10 suggests a low-complexity library and is worth investigating before you spend compute on calling.
MarkDuplicatesSpark does the same job and can also do the coordinate sort, which saves a pass over the data. It is less forgiving of memory pressure, so we default to the plain version unless the pipeline is running on a large machine.
4. Check contamination and coverage before calling
Running quality control before variant calling saves hours of work on a sample that was never going to produce a usable callset. Two checks matter most: cross-sample contamination and coverage uniformity.
mosdepth --by 1000 --fast-mode S1 S1.dedup.bam
gatk CollectWgsMetrics -I S1.dedup.bam -R $REF -O S1.wgs_metrics.txt \
--INTERVALS wgs_calling_regions.hg38.interval_list
For a nominal 30x PCR-free library, expect mean coverage near 30, with at least 90 percent of the callable genome at 20x or more. Contamination is the more dangerous failure, because a few percent of foreign DNA produces low-allele-fraction calls that look like real heterozygous sites at low depth. VerifyBamID2 reports a FREEMIX estimate, and we treat anything above 0.02 as a reason to stop and re-examine the sample. Below about 0.01 the effect on germline calling is small.
5. Recalibrate base qualities
Base quality scores from the instrument carry systematic biases that depend on machine cycle, sequencing context, and read group. Base Quality Score Recalibration (BQSR) fits an empirical error model at sites that are not known variants and rewrites the quality scores accordingly. The caller’s genotype likelihoods depend directly on those scores, so the correction propagates to every call.
gatk BaseRecalibrator \
-R $REF -I S1.dedup.bam \
--known-sites Homo_sapiens_assembly38.dbsnp138.vcf \
--known-sites Mills_and_1000G_gold_standard.indels.hg38.vcf.gz \
--known-sites Homo_sapiens_assembly38.known_indels.vcf.gz \
-O S1.recal.table
gatk ApplyBQSR \
-R $REF -I S1.dedup.bam --bqsr-recal-file S1.recal.table \
-O S1.bqsr.cram --create-output-bam-index true
Two details are easy to get wrong. The known-sites files must use the same contig names as your reference, or GATK will fail with a sequence dictionary mismatch. And ApplyBQSR quantizes output qualities into a small number of bins by default, which is intentional and shrinks the file considerably. Writing CRAM instead of BAM here cuts storage roughly three to four fold against the same reference.
6. Call variants with HaplotypeCaller in GVCF mode
HaplotypeCaller finds active regions, locally reassembles the reads into candidate haplotypes with a De Bruijn graph, and scores each read against each haplotype with a pair hidden Markov model. That reassembly step is why it handles indels and clustered variants far better than a position-by-position pileup caller 1. Emit a GVCF rather than a VCF, even for one sample, because the GVCF records confidence at non-variant positions. That lets you distinguish “reference” from “no data” later, which matters when someone asks whether a specific position was covered.
HaplotypeCaller is the slowest step and scales poorly with threads inside a single process. Scatter it across intervals instead:
gatk SplitIntervals -R $REF \
-L wgs_calling_regions.hg38.interval_list \
--scatter-count 50 -mode INTERVAL_SUBDIVISION -O intervals/
ls intervals/*.interval_list | parallel -j 12 \
'gatk --java-options "-Xmx6g -XX:+UseParallelGC -XX:ParallelGCThreads=2" HaplotypeCaller \
-R '"$REF"' -I S1.bqsr.cram \
-O gvcf/{/.}.g.vcf.gz \
-L {} -ERC GVCF \
-G StandardAnnotation -G AS_StandardAnnotation \
--native-pair-hmm-threads 2'
gatk MergeVcfs $(ls gvcf/*.g.vcf.gz | sed 's/^/-I /') -O S1.g.vcf.gz
Scattered 50 ways across 12 concurrent jobs, a 30x genome finishes in roughly two to four hours. Unscattered on one core it can run past a day. The AS_StandardAnnotation group adds allele-specific annotations, which give better filtering behavior at multiallelic sites and cost almost nothing to compute.
Note that wgs_calling_regions.hg38.interval_list excludes alt and decoy contigs. The decoys still do their job during alignment by attracting reads, but you do not want variant calls on them.
7. Genotype the GVCF
GenotypeGVCFs converts per-sample likelihoods into genotype calls with a posterior model. For one sample the prior is uninformative, so the main thing this step does is drop non-variant blocks and compute the final QUAL and genotype quality values.
gatk GenotypeGVCFs \
-R $REF -V S1.g.vcf.gz \
-O S1.raw.vcf.gz \
-G StandardAnnotation -G AS_StandardAnnotation \
--dbsnp Homo_sapiens_assembly38.dbsnp138.vcf
Expect roughly 4.5 to 5 million SNVs and 800,000 to 1,000,000 indels from a 30x genome against GRCh38. The exact number depends on ancestry, since distance from the reference drives the count. A result far outside that range is a signal that something upstream went wrong.
8. Filter the raw calls
Raw calls contain a large fraction of artifacts, and separating them from true variants is the step that most affects how usable the callset is 1. GATK’s Variant Quality Score Recalibration (VQSR) learns the boundary from the data itself by fitting a Gaussian mixture model to annotation values at known true sites. It needs enough variants to fit that model, which a single whole genome does supply for SNVs but not comfortably for indels.
For one sample we use hard filters. They are transparent, reproducible, and easy to relax when you want to look at a specific locus by hand.
gatk SelectVariants -V S1.raw.vcf.gz -select-type SNP -O S1.snp.vcf.gz
gatk SelectVariants -V S1.raw.vcf.gz -select-type INDEL -select-type MIXED -O S1.indel.vcf.gz
gatk VariantFiltration -V S1.snp.vcf.gz -O S1.snp.filt.vcf.gz \
-filter "QD < 2.0" --filter-name QD2 \
-filter "QUAL < 30.0" --filter-name QUAL30 \
-filter "SOR > 3.0" --filter-name SOR3 \
-filter "FS > 60.0" --filter-name FS60 \
-filter "MQ < 40.0" --filter-name MQ40 \
-filter "MQRankSum < -12.5" --filter-name MQRankSum-12.5 \
-filter "ReadPosRankSum < -8.0" --filter-name ReadPosRankSum-8
gatk VariantFiltration -V S1.indel.vcf.gz -O S1.indel.filt.vcf.gz \
-filter "QD < 2.0" --filter-name QD2 \
-filter "QUAL < 30.0" --filter-name QUAL30 \
-filter "FS > 200.0" --filter-name FS200 \
-filter "ReadPosRankSum < -20.0" --filter-name ReadPosRankSum-20
gatk MergeVcfs -I S1.snp.filt.vcf.gz -I S1.indel.filt.vcf.gz -O S1.filtered.vcf.gz
It helps to know what these annotations measure. The QD value is variant confidence divided by depth of the supporting reads, which stops deep sites from passing on depth alone. Both FS and SOR measure strand bias, the tendency for an allele to appear only on forward or only on reverse reads. The ReadPosRankSum test detects alleles that appear preferentially near read ends, a classic signature of alignment artifacts around indels.
These thresholds mark variants rather than delete them, so the FILTER column still tells you why something was excluded. Keep the marked records. When you later look at a specific gene, a filtered call with a near-threshold QD is worth inspecting in IGV rather than ignoring.
9. Validate the pipeline against a benchmark genome
Running your pipeline once on a Genome in a Bottle sample is the single most useful thing you can do to calibrate your trust in it. Download HG002 30x FASTQ files, run the identical pipeline, and compare against the v4.2.1 benchmark VCF and its high-confidence BED file.
hap.py HG002_benchmark.vcf.gz HG002.filtered.vcf.gz \
-f HG002_highconf.bed -r $REF -o happy_out --engine vcfeval
A well-configured GATK pipeline on 30x Illumina data typically lands above 0.99 F-measure for SNVs inside high-confidence regions, with indel performance several points lower. If your numbers are far below that, the problem is in the pipeline and not in your sample. This benchmark run also tells you which categories to distrust, and indels in homopolymer runs will be the weakest.
For your own sample you have no truth set, so use distributional checks instead:
bcftools stats -f PASS S1.filtered.vcf.gz | grep -E '^SN|^TSTV'
The transition/transversion ratio for whole-genome SNVs should sit near 2.0 to 2.1. A value near 1.0 means noise is dominating, since random errors produce transitions and transversions in roughly equal measure. Also check the fraction of passing SNVs present in dbSNP, which should exceed 95 percent for a genome of common ancestry.
10. Normalize and annotate
Before annotation, normalize representation. The same indel can be written several ways, and annotation tools will silently disagree with your variant if the representation differs.
bcftools norm -m -any -f $REF -Oz -o S1.norm.vcf.gz S1.filtered.vcf.gz
bcftools index -t S1.norm.vcf.gz
-m -any splits multiallelic records into one line per alternate allele, and -f left-aligns indels against the reference. Run this before any annotation step, and run it again on any external VCF you intend to intersect with your own. Then annotate with Ensembl VEP. It gives consequence predictions, gene and transcript context, and population frequencies.
vep -i S1.norm.vcf.gz -o S1.vep.vcf.gz --vcf --compress_output bgzip \
--cache --offline --assembly GRCh38 --everything --fork 8 \
--fasta $REF --pick_allele_gene
From here, bcftools query will flatten anything you want into a table for analysis:
bcftools +split-vep S1.vep.vcf.gz -f '%CHROM\t%POS\t%REF\t%ALT\t%GT\t%SYMBOL\t%Consequence\t%gnomADg_AF\n' -d -A tab
One point about what this callset is and is not. A research-grade pipeline like this produces hypotheses. Its output does not count as clinical findings. Any variant you intend to act on needs confirmation in a clinical laboratory and interpretation by a clinician or genetic counselor. Short-read WGS also misses structural variants, repeat expansions, and much of the variation in segmental duplications and the HLA region.
Common problems
Sequence dictionary mismatches cause more failed runs than anything else. GATK compares contig names and order between your reference, BAM, and every VCF resource. A file using chr1 against a reference using 1 fails immediately, and bcftools annotate --rename-chrs is the fix.
Missing or wrong read groups produce an error in BaseRecalibrator or silently degrade recalibration. If reads from four lanes share one read group, BQSR fits a single error model to four instruments’ worth of distinct behavior. Check with samtools view -H S1.bam | grep '^@RG' before you go further.
HaplotypeCaller running out of memory usually traces to a heap setting that ignores the parallel jobs around it. Twelve concurrent jobs at -Xmx6g require 72 GB of heap before you count the JVM overhead and the operating system’s page cache. Reduce concurrency rather than heap, because the pair HMM step is memory-hungry per job.
Very slow HaplotypeCaller runs often come from regions with extreme depth, such as centromeric or rDNA pileups. Using the standard WGS calling intervals avoids most of them. If a single shard still stalls, --max-reads-per-alignment-start 50 caps downsampling per position and usually resolves it.
Finally, a callset that looks plausible in aggregate can still be wrong at the one locus you care about. Before drawing a conclusion about a specific variant, open the CRAM in IGV at that position and look at the reads. Check the mapping qualities, the allele balance, and whether the supporting reads are all in the same orientation. That five-minute look catches errors that no summary metric surfaces 1.
Footnotes
-
Peter N. Robinson, Rosario M. Piro, Marten Jäger. Variant Calling and Quality-Based Filtering. Computational Exome and Genome Analysis, 2017. https://doi.org/10.1201/9781315154770-12 ↩ ↩2 ↩3
Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.