Data Dictionary

A data dictionary is a separate, machine-readable file that defines every field in a dataset: its name, type, units, permitted values, missingness convention, and how it was produced. It is the contract between the person who made the data and the person (or agent) who reads it six months later.

How it works

A useful dictionary has one row per field and, at minimum, these columns: field_name, label, data_type, units, permissible_values, missing_code, source_file, provenance. Two conventions dominate in practice.

The first is the tabular codebook: a CSV or spreadsheet, one row per variable, human-written. REDCap exports one, and most clinical registries use this shape. It is cheap and it is what most consortium efforts converge on after arguing about terminology. The ICGC ARGO dictionary is a good large-scale example: field definitions with types, controlled vocabularies, and validation rules that reject a submission before it enters the archive, developed with clinical input across many contributing centers.¹ A recent vascular-disease dictionary followed the same path, building consensus variable definitions specifically so downstream models train on fields that mean the same thing at every site.²

The second is the semantic dictionary: each field is bound to a term in an ontology (LOINC, UBERON, HGNC, UO for units) rather than to a prose sentence. This is what lets two datasets merge without a human reading both codebooks. The Semantic Data Dictionary approach formalizes this, annotating each column with entity, attribute, unit, and role so that a table becomes an RDF graph you can query.³ The older Medical Entities Dictionary made the same argument from clinical systems: concept-oriented terminologies, with explicit relationships between concepts, survive schema change in a way that string labels do not.⁴

We would write both. The CSV is what you read. The ontology bindings are what an agent reads, and they take an extra afternoon.

In your own data

Every file in a molecular profile already ships with a partial, implicit dictionary. Your job is to make it explicit and put it in one place.

• VCF: the header ##INFO and ##FORMAT lines are the dictionary. ##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read depth">. Pull them out with bcftools view -h sample.vcf.gz | grep '^##INFO'. Record which caller wrote them and which reference build (GRCh38 vs. hg19 coordinates differ, and a merged file with both is silently wrong).
• RNA-seq: a quant.sf from Salmon has Name, Length, EffectiveLength, TPM, NumReads. Your dictionary must state the annotation version (GENCODE v44, not “GENCODE”) because transcript IDs are versioned and ENST00000456328.2 will not join to ENST00000456328.1. State that TPM is within-sample normalized and cannot be compared across samples without re-normalization.
• Proteomics: an Olink or SomaScan matrix gives NPX or RFU, both relative and both log-scaled in most exports. Write down the log base. Write down the LOD-censoring rule, because values below LOD are often imputed rather than left null, and averaging imputed values is how you manufacture a trend.
• CGM: Dexcom exports mg/dL by default in the US and mmol/L elsewhere, with 5-minute cadence. Record the timezone of the timestamp column and whether DST transitions were normalized. Record calibration gaps as an explicit missing_code, not as zeros.

Common mistakes, in order of how often we see them: units absent; missing values encoded as -9, 999, NA, and empty string in the same file; column names that collide after a join (value, score, id); no reference build; no record of which pipeline version produced which file. Fix the last one by storing a provenance column holding the exact command line.

Keep the dictionary next to the data, version it in git, and validate against it. A short script that asserts every column in every file appears in the dictionary, and that types and ranges match, will catch the merge errors that otherwise surface as a biologically interesting result.

Limitations

A dictionary describes fields, not biology. It tells you a column holds TPM for ENSG00000141510; it will not tell you what that transcript’s abundance means for you. Gene-level annotation lives in separate compendia built for that purpose.⁵⁶

Controlled vocabularies also drift. Terms are deprecated, merged, and re-scoped, and a dictionary written against one release will decay against the next unless you pin the version. Automated extraction of definitions from free text helps but is imperfect: dictionary-based concept recognition in biomedical text still misses variants and novel terms, which is why hybrid neural approaches exist.⁷ Efforts to generate dictionary descriptions automatically for existing tables are promising and not yet a substitute for a person writing down what they measured.⁸

Finally, none of this is interpretation. A well-specified lab value is still a lab value. If a number sits outside a reference range and you want to know what it means for you, that is a conversation with a clinician.

Woolf Software builds longitudinal molecular profiles of individuals: whole-genome sequencing, RNA sequencing, proteomics, blood biomarkers, and continuous glucose data, integrated into one model of you. Build your profile.