Skip to content

How to Query the gnomAD v4.1 Sites Hail Table in Google Cloud

Oak
A feathered reptile-bird specimen on a black plinth, its flank striped in columns of translucent feathers with only a few lit.

By the end of this guide you will have a Hail Table of your own variants annotated with gnomAD v4.1 genome allele counts, filtered allele frequencies, and site filters. You will also have a much cheaper path for the common case where you only care about a few dozen genes. You need a Google Cloud project with billing enabled, the gcloud CLI, and Python 3.9 or newer. You also need Hail 0.2. We pin a single version per project and use hailctl dataproc so the cluster matches the client. You also need your own variants in a GRCh38-aligned VCF. If your VCF is on GRCh37, stop and either recall against GRCh38 or lift over before you go further, because the v4.1 tables exist only on GRCh38.

1. Inventory the bucket and pick the right table

The gnomAD release lives in a US multi-region Google Cloud Storage bucket named gcp-public-data--gnomad. Start by listing what is there rather than copying a path from a blog post, because the directory layout has changed between major releases.

gcloud storage ls gs://gcp-public-data--gnomad/release/4.1/ht/
# .../release/4.1/ht/exomes/
# .../release/4.1/ht/genomes/
# .../release/4.1/ht/joint/

gcloud storage ls gs://gcp-public-data--gnomad/release/4.1/ht/genomes/

Three sites tables matter for allele frequency work. The genomes table is gs://gcp-public-data--gnomad/release/4.1/ht/genomes/gnomad.genomes.v4.1.sites.ht and covers 76,215 whole genomes. The exomes table covers 730,947 exomes and gives you far more chromosomes of evidence inside coding regions. The joint table combines both and is the one you want when you need a single allele number across everything that was called at a site.

Which one to use depends on where your variant is. For coding and splice-region variants we use the exomes or joint table, because the allele number is an order of magnitude larger and rare-variant frequency estimates are correspondingly tighter. For deep intronic, promoter, and intergenic variants the genomes table is the only one with real coverage. The joint table is also where v4.1 added the flag for sites whose exome and genome frequencies disagree strongly, which is a useful signal that one of the two callsets has a technical problem at that position.

Before you plan any job, measure the object you are about to read.

gcloud storage du -s gs://gcp-public-data--gnomad/release/4.1/ht/genomes/gnomad.genomes.v4.1.sites.ht

This is a multi-terabyte Hail Table stored as partitioned Parquet-like files with a row index. You should never plan to read all of it unless you have a genuine whole-genome join to do. Keep your compute in the US to sit next to the data and avoid cross-region transfer.

2. Read the schema before you write a query

Field names drift between releases, and guessing them is the most common way to waste an hour. Open the table and print the schema first. Interval queries and describe() work fine from a laptop, because Hail ships the GCS connector and only touches the partitions you ask for.

import hail as hl

hl.init(
    default_reference="GRCh38",
    tmp_dir="gs://my-scratch-bucket/hail-tmp",
)

GENOMES = "gs://gcp-public-data--gnomad/release/4.1/ht/genomes/gnomad.genomes.v4.1.sites.ht"

ht = hl.read_table(GENOMES)
ht.describe()
print(ht.n_partitions())

The table is keyed by locus (a LocusExpression on GRCh38) and alleles (an array of two strings, reference then alternate). Every variant is already split to one alternate allele per row, so you do not need to split multiallelics on the gnomAD side. Among the row fields you will find the following.

  • freq, an array of structs with AC, AF, AN, and homozygote_count in each
  • faf, the filtering allele frequencies
  • filters, a set of strings such as AC0 and the variant-quality-score filters
  • vep, the full VEP annotation struct
  • region_flags for low-complexity and segmental-duplication overlaps
  • in_silico_predictors with precomputed scores

The globals carry the index dictionaries that make freq and faf readable.

Read the output of describe() rather than my description of it. Treat the schema as the source of truth, and write your code against the names you see.

3. Query one gene without reading the table

Hail prunes partitions when you filter on the key with intervals, so a single-gene query over a multi-terabyte table costs seconds and a few hundred megabytes of reads. This is the single most valuable technique for working with gnomAD interactively.

intervals = [
    hl.parse_locus_interval("chr13:32310000-32405000", reference_genome="GRCh38"),  # BRCA2, padded
]
brca2 = hl.filter_intervals(ht, intervals)
print(brca2.count())

Two details matter. Use filter_intervals and not ht.filter(ht.locus.contig == "chr13"), because the latter is a row-level predicate that still scans every partition. And pad your gene coordinates by a kilobase or two on each side so you capture splice regions and the UTR boundaries that gene models disagree about.

If you have a list of positions rather than a range, build the interval list programmatically from a BED file with hl.import_bed, then pass ht.filter(hl.is_defined(bed[ht.locus])) after an interval filter on the same regions. The interval filter does the partition pruning and the join does the exact selection.

4. Decode the frequency arrays

freq is a flat array, and the meaning of each position lives in the freq_index_dict global. Evaluate the dictionary once, then index by name.

fid = hl.eval(ht.freq_index_dict)
print(len(fid), list(fid)[:20])

ann = brca2.annotate(
    AC_adj=brca2.freq[fid["adj"]].AC,
    AN_adj=brca2.freq[fid["adj"]].AN,
    AF_adj=brca2.freq[fid["adj"]].AF,
    nhomalt_adj=brca2.freq[fid["adj"]].homozygote_count,
)

The adj entry is the overall frequency after genotype-quality adjustment, and this is the number you almost always want. The raw entry includes low-quality genotypes and will give you inflated counts for borderline sites. Genetic-ancestry-group entries follow the pattern nfe_adj, afr_adj, and sas_adj. Sex-stratified keys such as nfe_XX_adj follow the same pattern. Print the keys and use exactly what is there.

For rare-disease interpretation the filtering allele frequency is more useful than the point estimate. It is the lower bound of a confidence interval on the highest group-specific frequency, so a variant with fafmax above your threshold is one you can set aside with statistical justification rather than a hunch.

faf_idx = hl.eval(ht.faf_index_dict)
ann = ann.annotate(faf95_adj=ann.faf[faf_idx["adj"]].faf95)
ann.select("AC_adj", "AN_adj", "AF_adj", "faf95_adj", "filters").show(20)

Always carry AN alongside AF. A variant with AC=1 and AN=200 is not rare in any meaningful sense, it is unmeasured. Low allele number at a site is the standard explanation for surprising frequencies, and the v4.1 release exists in part because allele numbers in v4.0 were wrong at sites with no coverage.

5. Annotate your own variants

Now join your own callset. For a single person’s whole genome you are looking at four to five million variants, which is small on the query side but forces a real read of the gnomAD table if you do it naively. Two approaches work, and they trade compute for setup effort.

The straightforward version runs on a Dataproc cluster. Start it with hailctl so the Spark and Hail versions are consistent.

hailctl dataproc start gnomad-anno \
  --region us-central1 \
  --num-workers 2 \
  --num-secondary-workers 20 \
  --worker-machine-type n1-standard-8 \
  --max-idle 30m

Then submit a script that imports your VCF, normalizes it, and joins.

import hail as hl

hl.init(default_reference="GRCh38")

mt = hl.import_vcf(
    "gs://my-scratch-bucket/me.g.vcf.bgz",
    reference_genome="GRCh38",
    force_bgz=True,
    array_elements_required=False,
)
mt = hl.split_multi_hts(mt)
mt = mt.key_rows_by(
    locus=mt.locus,
    alleles=[hl.min_rep(mt.locus, mt.alleles).alleles[0],
             hl.min_rep(mt.locus, mt.alleles).alleles[1]],
)

g = hl.read_table(
    "gs://gcp-public-data--gnomad/release/4.1/ht/genomes/gnomad.genomes.v4.1.sites.ht"
)
fid = hl.eval(g.freq_index_dict)

g = g.select(
    gnomad_AC=g.freq[fid["adj"]].AC,
    gnomad_AN=g.freq[fid["adj"]].AN,
    gnomad_AF=g.freq[fid["adj"]].AF,
    gnomad_nhomalt=g.freq[fid["adj"]].homozygote_count,
    gnomad_filters=g.filters,
    gnomad_most_severe=g.vep.most_severe_consequence,
)

mt = mt.annotate_rows(gnomad=g[mt.row_key])
mt.rows().write("gs://my-scratch-bucket/me.gnomad_v4.1.ht", overwrite=True)

Two things in that script are load-bearing. The select on the gnomAD table happens before the join, so Hail only reads the columns you asked for instead of the whole VEP struct and all the histograms. And the left-normalization step matters, because gnomAD stores minimal representations of indels. An unnormalized indel from your caller will silently fail to match and come back with a missing annotation, which reads as “absent from gnomAD” and is the most dangerous failure mode in this whole pipeline.

Run hailctl dataproc stop gnomad-anno when the job finishes. Secondary (preemptible) workers make this affordable, and a whole-genome join on twenty of them is on the order of an hour rather than a day.

6. The lightweight path for a gene panel

If your question is about a specific set of genes, skip Spark entirely. The release also ships per-chromosome sites VCFs, and htslib can read them over HTTPS using the remote tabix index.

BASE=https://storage.googleapis.com/gcp-public-data--gnomad/release/4.1/vcf/genomes

# Inspect the INFO fields that are available
bcftools view -h $BASE/gnomad.genomes.v4.1.sites.chr13.vcf.bgz | grep '^##INFO' | head -40

# Pull one gene
bcftools view -r chr13:32310000-32405000 \
  $BASE/gnomad.genomes.v4.1.sites.chr13.vcf.bgz \
  -Oz -o gnomad_brca2.vcf.gz
bcftools index -t gnomad_brca2.vcf.gz

Then transfer the fields you want onto your own VCF. Normalize both sides against the same reference FASTA first, for exactly the indel-matching reason above.

bcftools norm -f GRCh38.primary.fa -m -any -Oz -o me.norm.vcf.gz me.vcf.gz
bcftools index -t me.norm.vcf.gz

bcftools annotate \
  -a gnomad_brca2.vcf.gz \
  -c INFO/AC,INFO/AN,INFO/AF,INFO/nhomalt,INFO/AF_grpmax,INFO/faf95_max \
  -Oz -o me.annotated.vcf.gz me.norm.vcf.gz

Check the exact INFO tag names in the header output above and adjust -c to match. This path runs on a laptop in minutes for a panel of a few hundred genes. It stops scaling somewhere around a few megabases of query regions, at which point the Hail route is cheaper.

7. Interpreting what comes back

The annotations are measurements, and they have known limits. Treat a missing gnomAD entry as weak evidence and not as proof of novelty, because the site may be inside a region where the callset has poor coverage. Check region_flags and the allele number before you conclude anything about rarity.

Population-level frequency estimates from this table support useful downstream calculations. Carrier frequencies for autosomal and X-linked recessive conditions have been estimated directly from gnomAD v4.0 across ancestry groups, which gives you a sense of what the frequency fields can carry when applied carefully 1. The published correspondence on that analysis is worth reading alongside it, because it lays out where gene-level and variant-classification choices change the answer substantially 2. Ancestry group labels in gnomAD are genetic-ancestry groupings and not social categories, and genetic variation that tracks with ancestry propagates into other molecular layers such as DNA methylation 3.

Two further capabilities are easy to miss. gnomAD publishes variant co-occurrence data that indicates whether two variants in the same gene tend to appear on the same haplotype, which has been evaluated against real trio-phased cases in autosomal recessive disorders 4. And for questions that need genotypes rather than summary counts, the harmonized HGDP and 1000 Genomes callset released alongside gnomAD provides deeply sequenced individual-level data across diverse populations 5.

Frequency alone will not tell you whether a variant matters. Noncoding and synonymous variants need their own models, and gnomAD frequencies are an input to those models rather than a substitute. Machine learning models for synonymous variant pathogenicity are trained partly on gnomAD-derived constraint signals 6. Constraint at the level of protein-RNA interfaces has been mapped by combining binding data with population variation, which shows how far beyond coding consequence this kind of analysis now reaches 7. Nothing you compute from these tables is a diagnosis. If an annotated variant looks clinically relevant, take it to a clinical geneticist or a genetic counselor for validation and classification.

Common problems

Missing annotations for indels almost always mean a representation mismatch. Normalize your VCF with bcftools norm -f <reference> and confirm that a known common indel in your callset matches a gnomAD row before you trust the whole join.

Contig naming mismatches produce empty joins with no error. gnomAD v4.1 uses chr1 style names on GRCh38. If your VCF uses 1, rename with bcftools annotate --rename-chrs before importing.

Jobs that run for hours on a single gene usually mean you filtered with a row predicate instead of hl.filter_intervals. Check the query plan by looking at whether partition count drops after your filter.

Out-of-memory failures in Hail during the join are typically caused by carrying the full vep struct or the histogram arrays through a shuffle. Apply select to the gnomAD table first and keep only scalar fields.

A raw frequency that looks much higher than the adj frequency means low-quality genotypes are driving the count. Use adj and check filters for AC0, which marks sites where no high-quality genotype survived.

Structural and copy-number variants live in separate SV releases, and the sites tables do not hold them. Mosaic events in particular need dedicated detection from genome sequencing rather than a frequency lookup 8.

Finally, cost surprises come from leaving clusters running. Set --max-idle on every hailctl dataproc start and confirm teardown with gcloud dataproc clusters list --region us-central1.

Footnotes

  1. Ronja Hotakainen, Timo Järvinen, Kaisa Kettunen, et al. Estimation of carrier frequencies of autosomal and X-linked recessive genetic conditions based on gnomAD v4.0 data in different ancestries. Genetics in Medicine, 2025. https://doi.org/10.1016/j.gim.2024.101304 ↩

  2. Mia J. Gruzin, Swaroop Aradhya, Leslie Burnett. Correspondence on “Estimation of carrier frequencies of autosomal and X-linked recessive genetic conditions based on gnomAD v4.0 data in different ancestries” by Hotakainen et al. Genetics in Medicine, 2025. https://doi.org/10.1016/j.gim.2025.101559 ↩

  3. Kyriaki Founta, Nyasha Chambwe. Genetic variation underlies ancestry-associated DNA methylation differences with immune regulatory potential in breast tumors. Communications Biology, 2026. https://doi.org/10.1038/s42003-026-10784-x ↩

  4. Jaeryuk Kim, Gu-Hwan Kim, Sunghee Min, et al. Real-world evaluation of gnomAD variant co-occurrence information for haplotype phasing in autosomal recessive disorders. Human Genomics, 2025. https://doi.org/10.1186/s40246-025-00855-1 ↩

  5. Zan Koenig, Mary T. Yohannes, Lethukuthula L. Nkambule, et al. A harmonized public resource of deeply sequenced diverse human genomes. Genome Research, 2024. https://doi.org/10.1101/gr.278378.123 ↩

  6. Linnaeus Bundalian, Martina Schmidt Strnadová, Felix Garten, et al. SyMetrics: an integrated machine learning model for evaluating the pathogenicity of synonymous variants in the human genome. NAR Genomics and Bioinformatics, 2026. https://doi.org/10.1093/nargab/lqaf211 ↩

  7. Hsuan-lin Her, Brian A. Yee, Shuhao Xu, et al. Comprehensive RNA-binding protein analyses and deep learning uncover genetic constraints and disease associations in protein-RNA interfaces. Cell Systems, 2026. https://doi.org/10.1016/j.cels.2026.101588 ↩

  8. Naomi Tsuchida, Yuri Uchiyama, Kohei Hamanaka, et al. Mosaic deletions detected by genome sequencing in two families. Journal of Human Genetics, 2025. https://doi.org/10.1038/s10038-025-01336-y ↩

Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.