Parquet

Parquet is a binary columnar storage format: values from the same column are stored contiguously, compressed per column, and annotated with min/max statistics per chunk so a reader can skip most of the file for most queries. It is not a table in the SQL sense and not a database. It is a file, or a directory of files, that engines like DuckDB, Polars, pandas, Spark, and Arrow read natively.

How it works

A Parquet file is split into row groups (typically 64–512 MB of uncompressed data each). Within a row group, each column is stored as a column chunk, and each chunk is split into pages (default 1 MB). The footer at the end of the file holds the schema and, per column chunk, min, max, null count, and byte offsets.

Two things follow from that layout. Column pruning: reading POS and AF from a 40-column variant table touches only two column chunks. Predicate pushdown: a filter like POS BETWEEN 1000000 AND 1100000 lets the reader consult footer statistics and skip whole row groups. This only works if the data is sorted on the filter column. Parquet does not sort for you.

Compression is per column, which is why it beats gzipped CSV. A CHROM column with 25 distinct values gets dictionary-encoded to a few bits per row. Sorted integer positions get delta-encoded. A GT column of 0/0, 0/1, 1/1 collapses to almost nothing. Text formats interleave these columns row by row, so the compressor never sees the redundancy clearly.

Answering the obvious comparisons directly. Against CSV: same logical content, different physical layout, plus a real schema, so chr1 stays a string and 0.0001 stays a double instead of being re-guessed on every load. Against JSON: Parquet handles nested structures (lists, structs, maps) but stores them shredded into flat columns with definition and repetition levels, so nesting costs almost nothing to query. Lighter than CSV: yes, typically 5–20× smaller for genomic tables, and one report on VCF data puts the compression factor around 10.

In your own data

Start with your variant calls. A single-sample WGS VCF is roughly 4–5 million variants and 1–2 GB gzipped. Convert with DuckDB or with bcftools query piped into a Parquet writer:

COPY (
  SELECT * FROM read_csv('variants.tsv', delim='\t', header=true)
  ORDER BY chrom, pos
) TO 'variants.parquet'
  (FORMAT PARQUET, COMPRESSION ZSTD, ROW_GROUP_SIZE 1000000);

Sort by chrom, pos before writing. This is the single choice that determines whether region queries take 30 ms or 3 seconds. Use ZSTD level 3 over Snappy unless you are CPU-bound on reads: roughly 20–30% smaller for a small decompression cost. Partition by chromosome (hive_partitioning) if you routinely query one chromosome at a time, but do not partition on anything high-cardinality. Thousands of tiny files are slower than one big one.

The same pattern applies beyond variants. RNA-seq count matrices from a salmon/tximport run, Olink or SomaScan NPX tables, CGM exports at 5-minute resolution (about 105,000 rows per year per sensor), and lab panels all become one Parquet file each with a shared sample_id and timestamp. Then a join across assays is a single DuckDB query, no ETL framework. Selective-execution work on population variant sets shows the same idea at scale: skipping non-relevant partitions of a columnar store is where the speedup comes from, not faster arithmetic.¹ Big-data variant pipelines built on Hadoop and Spark made the same bet a decade earlier.² Mass spectrometry is moving the same direction, with proposed successors to mzML built on Parquet for exactly these properties.³

Common mistakes, in the order we have hit them. Writing unsorted data and wondering why pushdown does nothing. Row groups of 100 rows, which makes the footer larger than the data. Storing genotype strings instead of encoded integers. Letting a reader infer chrom as an integer, which silently breaks on X, Y, and chrM. Assuming Parquet files are appendable: they are not, you write a new file into the dataset directory.

Limitations

Parquet is immutable and write-once. Correcting one genotype means rewriting a row group, so keep the original VCF or BAM as the archival source of truth and treat Parquet as a derived query layer. It is poor for random single-row lookups by key: use a real index or a key-value store for that. It also flattens badly for genuinely non-tabular structure. Hypergraph or multi-way contact representations of chromatin do not reduce to rows and columns without loss.⁴ And nothing in Parquet is clinical. A fast query over your variants tells you what a caller reported, not what it means for you. Interpretation of any specific variant belongs with a clinical geneticist or genetic counselor.

Woolf Software builds longitudinal molecular profiles of individuals: whole-genome sequencing, RNA sequencing, proteomics, blood biomarkers, and continuous glucose data, integrated into one model of you. Build your profile.