Missingness

Missingness is the structure of what your data does not contain: which values are absent, why they are absent, and whether the reason is related to the value itself. It is a property of the measurement process, not a nuisance to be cleaned away, and the mechanism decides which repairs are legitimate.

The standard taxonomy has three classes. Missing completely at random (MCAR): absence is independent of everything, like a tube dropped on the floor. Missing at random (MAR): absence depends on observed variables, like a panel that failed for every sample run on one plate, where plate ID is recorded. Missing not at random (MNAR): absence depends on the unobserved value itself, like a protein whose concentration fell below the instrument’s detection limit. MCAR and MAR can be imputed with standard machinery. MNAR cannot be imputed by any method that assumes the missing values look like the observed ones, because they systematically do not.

How it works

Genotype imputation is the one case where missingness is genuinely solved, and it is solved because the underlying biology supplies a reference. Haplotypes are shared, so an untyped variant can be inferred from flanking typed variants using a hidden Markov model over a reference panel of phased haplotypes. This is the method behind essentially every GWAS since 2007 ¹. Accuracy scales with panel size and ancestry match: common variants (MAF > 5%) reach r² above 0.95 with modern panels, while variants below 0.5% degrade sharply and degrade faster for ancestries underrepresented in the panel ²³.

Everything else is weaker. In transcriptomics and proteomics you have no reference panel, so imputation borrows structure from correlations across features and samples. The families are: single-value fills (mean, median, k-nearest neighbors), model-based methods (multiple imputation, low-rank matrix completion, MICE), and learned methods (autoencoders, VAEs, GANs), which outperform simple fills mainly when the missingness is high-dimensional and structured ⁴⁵. A useful distinction is element-wise missingness (scattered holes in one matrix) versus block-wise missingness (an entire assay absent for a subset of samples). Block missingness needs methods built for it, such as k-nearest-neighbor weighting across a complete companion omic ⁶. Some cross-omic prediction works well enough to be used deliberately: NMR metabolite profiles can serve as surrogates for several standard clinical risk factors ⁷.

For metabolomics and proteomics the productive move is to classify each feature’s mechanism first and then apply a matching imputation, rather than running one algorithm across the whole matrix. Two-step mechanism-aware approaches do exactly this and beat mechanism-blind imputation on downstream inference ⁸.

In your own data

Where the holes live, file by file:

• VCF from WGS. ./. genotypes are no-calls. Check the DP and GQ fields, not just the presence of a call. Filter with bcftools view -i 'FMT/DP>=10 & FMT/GQ>=20' and count what you lose. No-calls cluster in low-complexity regions, segmental duplications, and the HLA and pharmacogene loci (CYP2D6 especially). If a variant you care about is a no-call, the answer is a targeted assay, not imputation. Imputed dosages carry an INFO/R2 score in the VCF: treat anything below 0.8 as a soft call and below 0.3 as absent.
• RNA-seq counts. A zero in a count matrix is ambiguous between “not expressed” and “not captured.” For bulk RNA-seq at 30M+ reads, treat zeros as real low expression and filter (filterByExpr in edgeR, or CPM > 1 in at least a third of samples) rather than impute. Imputing bulk zeros inflates correlations.
• Proteomics. Mass-spec missingness is dominated by the detection limit, so it is MNAR. Plot detection frequency against mean intensity across your timepoints: if the low-abundance proteins are the missing ones, the curve is the proof. Left-censored imputation (a draw from a downshifted distribution, QRILC or MinProb) is defensible. Mean imputation is not.
• CGM. Gaps come from sensor warmup, compression during sleep, and transmission loss. Linear interpolation over five minutes is fine. Interpolating a two-hour gap and then computing time-in-range silently fabricates the answer. Report percent coverage alongside every derived metric.

The common mistake is imputing before quality control, so a systematically bad batch gets smoothed into plausibility. Do QC, drop features that exceed your missingness threshold, then impute.

How much is too much? There is no universal cutoff, but for element-wise MCAR/MAR data, imputation is usually stable to roughly 20-30% per feature, degrades noticeably past that, and should not be trusted above 50% ⁹. For MNAR data the acceptable fraction is lower, and for block missingness the question is not the percentage but whether a correlated complete assay exists.

Limitations

Imputation is a variance-reduction trick that trades bias for completeness. Single imputation understates uncertainty because the filled value is treated as observed, which is why multiple imputation exists and why any downstream confidence interval computed on singly imputed data is too narrow ⁴. Deep methods add a further risk: they can reproduce the dominant covariance structure so faithfully that imputed values look like the training distribution rather than like you ⁵. For genotypes specifically, imputation accuracy for you personally depends on how well the reference panel covers your ancestry ³.

If an imputed or absent value is the basis for a health decision, that is a conversation with a clinician working from a fresh targeted measurement, not from a filled cell in a matrix.

Woolf Software builds longitudinal molecular profiles of individuals: whole-genome sequencing, RNA sequencing, proteomics, blood biomarkers, and continuous glucose data, integrated into one model of you. Build your profile.