Feature Engineering

Feature engineering is the step where raw files become a matrix: rows are observations, columns are numbers with defined units and a defined reference, and every column exists because you decided it should. In molecular data this is most of the modeling work. A VCF with 4.5 million variants, a transcript quantification with 60,000 rows, and a proteomics panel with 3,000 analytes are not features. They are raw measurements at different scales, with different noise structures, on different sample sets.

How it works

Three broad model families take these matrices: supervised (predict a label), unsupervised (find structure without labels), and reinforcement learning (learn from sequential feedback). Almost everything in personal omics is the first two. Feature engineering feeds both.

The pipeline has four moves, in order:

1. Normalize. Counts become CPM, TPM, or DESeq2 median-of-ratios size factors. Proteomics NPX or intensity values get log2’d and median-centered per run. CGM gets resampled to a fixed grid (5-minute, linear interpolation for gaps under 20 minutes, drop longer gaps rather than filling them).
2. Transform. log1p on counts, rank-inverse-normal on skewed biomarkers, arcsinh when zeros matter and you want a shrinkage-friendly scale.
3. Aggregate. This is where biology enters. Transcript-level TPM summed to gene level with tximport(txOut=FALSE). Gene-level expression collapsed to pathway scores with GSVA or ssGSEA, taking 20,000 columns to ~50 Hallmark scores. Variants collapsed to gene-level burden (count of rare, high-impact alleles per gene) or to a polygenic score (a weighted sum over effect sizes, which is feature engineering with published weights).
4. Select. Filter by variance, then by a supervised criterion inside cross-validation. Benchmarks on metabolomics classification show the choice of selection and extraction method changes downstream accuracy substantially, and no single method dominates across datasets.¹ SVM-RFE is a reasonable default on multi-omics when sample count is small and you want a ranked list.²

Feature extraction differs from selection: extraction builds new coordinates (PCA, autoencoder bottleneck, embeddings from a pretrained model), selection keeps a subset of the originals.³ Extraction usually wins on accuracy and loses on interpretability. Genomic language models push this further: DNA sequence goes in, a learned embedding comes out, and that embedding replaces hand-built features like GC content and motif counts.⁴ They work for tasks like predicting enhancer activity and allele-specific effects directly from sequence.⁵ Foundation-model embeddings across modalities have improved drug response prediction in IBD over hand-built multi-omics features.⁶

In your own data

Feature engineering is not ETL. ETL moves bytes and enforces schemas. Feature engineering makes modeling decisions that leak information if you get the ordering wrong. Nor is it EDA: exploratory analysis is how you find out that your CGM has a 40-hour sensor gap, feature engineering is what you do about it.

Where it shows up in your files:

• variants.vcf.gz → annotate with VEP or SnpEff, filter to PASS, GQ >= 20, DP >= 10. Gene-level burden only over variants with a defined impact class. Population frequency thresholds matter more than people expect. Variant interpretation from a raw VCF remains the rate-limiting step in germline diagnosis, and it belongs with a clinical geneticist, not a script.⁷
• quant.sf (salmon) → tximport → DESeq2 vst(). Use vst, not raw TPM, when you plan to compute correlations or distances across timepoints.
• Proteomics NPX matrices → check the LOD column. Values below LOD are censored, not zero. Imputing them with zero after log-transform creates a bimodal column that a tree model will happily split on and learn nothing real.
• CGM CSV → engineered features are the useful part: time-in-range, MAGE, CONGA-1, postprandial AUC over 120 minutes from a timestamped meal log. Raw 5-minute values as 288 columns per day is almost never the right representation.

The mistake that costs the most: selecting features on the full dataset, then cross-validating. Selection must happen inside each training fold. Do it outside and your held-out AUC will be inflated, sometimes by 0.15 or more, and you will not find out until new data arrives.

Limitations

Engineered features encode assumptions. Pathway scores assume the pathway database applies to you. Polygenic scores assume the discovery cohort resembles you, and they transfer poorly across ancestries. Omics prognostic models show measurable performance disparities across ethnic groups traceable to the feature and training pipeline.⁸ Feature choice also constrains what the model can ever say: a model over pathway scores cannot implicate a single gene. Interpretability methods like SHAP attribute to whatever columns you built, so the attributions are only as meaningful as the aggregation.⁹ And with n=1 longitudinal data, most supervised setups are underpowered. Unsupervised structure and within-person change over time are the use cases.¹⁰

Any feature that looks clinically meaningful needs a clinician to interpret.

Woolf Software builds longitudinal molecular profiles of individuals: whole-genome sequencing, RNA sequencing, proteomics, blood biomarkers, and continuous glucose data, integrated into one model of you. Build your profile.