Provenance

Provenance is the full recorded history of a data file: which biological sample it came from, when and how that sample was collected and stored, which instrument and chemistry produced the signal, and every command, tool version, parameter, and reference file used to turn that signal into the file in front of you. Origin is one field in that record. Lineage is the graph of transformations. Provenance is both, plus enough detail to re-run the work and get the same answer.

The distinction people ask about is narrower than it sounds. Lineage answers “what produced this file, and what did that file produce?” It is a directed graph of inputs and outputs. Provenance answers the same question and adds the context that makes the answer usable: the aligner version, the --flags, the reference build and its patch level, the barcode-to-sample mapping, the freezer time between draw and extraction. Lineage tells you a VCF came from a BAM. Provenance tells you the BAM was aligned with bwa-mem2 2.2.1 against GRCh38.p13 with -K 100000000 and that the VCF was called with DeepVariant 1.6.1 in WGS mode.

How it works

In practice provenance is stored in three places, and you want all three.

Inside file headers. SAM/BAM/CRAM carry @PG lines, one per program that touched the file, each with ID, PN, VN, CL (the full command line), and PP linking to the previous program. @RG lines carry read group, sample, library, and platform unit. VCF headers carry ##reference=, ##source=, ##contig= lines with lengths and md5 values, and ##commandline= entries. Run samtools view -H sample.cram | grep '^@PG' and you should be able to reconstruct the whole pipeline.

Alongside the files. Checksums (md5sum, or b3sum if you want speed), a manifest, and a workflow description. Nextflow writes execution_trace.txt and execution_report.html per run. Snakemake writes .snakemake/metadata. Both are more useful than a README you maintain by hand.

In the container. If your pipeline runs in a pinned image (quay.io/biocontainers/bwa-mem2:2.2.1--hd03093a_2), the digest of that image is a provenance record for every tool version inside it. Pin by digest, not by tag. Tags move.

The formal versions of this exist: W3C PROV, RO-Crate, and the FAIR data guidelines that the community has been building out for shared genomic and outbreak data ¹. Regulatory work on computational evidence makes the same requirement explicit: a model result is only evidence if the inputs, code, and configuration behind it are traceable ².

In your own data

Concrete checks on a personal dataset, in the order we would do them.

1. Confirm the reference build. grep '##reference' variants.vcf and samtools view -H aln.cram | grep '^@SQ' | head. A chr1 with length 248956422 is GRCh38. 249250621 is GRCh37. Mixing them silently shifts every coordinate you look up by megabases.
2. Confirm the sample is yours. Run somalier extract on the CRAM and somalier relate against any other file that claims to be you. Genotype concordance across shared sites should be near 1.0. Sample swaps in sequencing centers are rare but not zero, and they are invisible without this check.
3. Confirm sex chromosomes and mitochondrial contig naming match between your BAM and your annotation files. chrM versus MT breaks joins quietly.
4. Record the annotation database versions. A VEP or ClinVar annotation is a snapshot. ClinVar 2024-03 and ClinVar 2025-09 disagree on thousands of variant classifications, mostly reclassifications toward benign. If you did not write down which release you used, your annotated VCF is not reproducible. This is exactly the curation-drift problem that biocuration teams work with directly ³.
5. For RNA-seq, keep the transcriptome FASTA and the GTF version next to the counts matrix. GENCODE v39 and v45 differ in gene models. A gene-level count is not comparable across them without re-quantification.

Common mistakes: rewriting a BAM with a tool that drops @PG history, renaming files as a substitute for metadata, storing provenance only in a notebook cell that gets re-executed, and keeping one “latest” copy of a VCF that you keep overwriting. Treat raw data as immutable and write every derived artifact to a new path with a run ID.

Limitations

Provenance records what the pipeline did, not whether the sample was any good. A CRAM can carry a perfect @PG chain and still come from a tube that sat at room temperature for six hours, degrading RNA integrity in ways no header records. Pre-analytical metadata (draw time, tube type, fasting state, processing delay) is the part most often missing and the part that most often explains a weird result.

Provenance also does not settle who may use the data. Consent, sharing terms, and downstream reuse are governance questions that ride on top of provenance records rather than being answered by them ⁴, and the sovereignty frameworks built for Indigenous genomic data exist precisely because a lineage graph says nothing about permission ⁵. If a provenance-traced result looks clinically significant, that is a conversation with a clinician, not a conclusion you draw from a header line.

Woolf Software builds longitudinal molecular profiles of individuals: whole-genome sequencing, RNA sequencing, proteomics, blood biomarkers, and continuous glucose data, integrated into one model of you. Build your profile.