How to Run Functional Enrichment Analysis on Your Own RNA-seq and Proteomics Data
By the end of this guide you will have three artifacts from your own data: a GSEA table with normalized enrichment scores and leading-edge genes for each significant gene set, a deduplicated over-representation result you can use as a sanity check, and a per-sample pathway score matrix you can plot against time. You need a count matrix from RNA sequencing (genes by samples), a sample metadata table, and R 4.3 or later. You also need the following packages.
DESeq2fgseamsigdbrclusterProfilerorg.Hs.eg.dbGSVA
If you also have proteomics, you need the intensity matrix plus the list of every protein the assay attempted to measure, which matters more than most people expect. Enrichment is the interpretation layer that sits between a differential abundance table and a shortlist of candidates, and it is the step in published multi-omics pipelines where a few thousand statistics become a handful of named biological programs 12.
1. Decide which question you are asking before choosing a method
There are two families of methods and they answer different questions. Over-representation analysis (ORA) takes a list of genes you already called significant and asks whether any gene set appears in that list more often than chance, given a background. Gene set enrichment analysis (GSEA) takes every gene you measured, ranked by a test statistic, and asks whether the members of a gene set sit toward one end of that ranking.
We use GSEA as the primary analysis and ORA as a cross-check. GSEA does not require a significance cutoff, so it does not throw away the coordinated small effects that dominate most human data. ORA is sensitive to where you set the cutoff and to a background choice that people routinely get wrong. The two agreeing is reassuring. The two disagreeing usually means your cutoff is doing the work.
2. Produce a ranked gene list from DESeq2
Enrichment quality is bounded by the quality of the statistics you feed it. Fit the model with the covariates you know, including sequencing batch and draw date, because a single person sampled over a year will show batch structure that mimics biology.
library(DESeq2)
dds <- DESeqDataSetFromMatrix(counts, coldata, ~ batch + condition)
keep <- rowSums(counts(dds) >= 10) >= 3
dds <- dds[keep, ]
dds <- DESeq(dds)
res <- results(dds, contrast = c("condition", "post", "pre"))
For ranking, use the Wald statistic in res$stat. Do not rank by the log2 fold change. The statistic is monotone in evidence and already accounts for dispersion, so a gene with a large fold change and three counts does not float to the top. Use lfcShrink(dds, coef = ..., type = "apeglm") when you want effect sizes to report, and keep it separate from the ranking vector.
Then clean the identifiers. Ensembl gene IDs from most pipelines carry a version suffix that will silently fail to join against gene set databases.
res_df <- as.data.frame(res)
res_df$gene <- sub("\\..*$", "", rownames(res_df))
res_df <- res_df[!is.na(res_df$stat), ]
res_df <- res_df[order(-abs(res_df$stat)), ]
res_df <- res_df[!duplicated(res_df$gene), ] # keep strongest per ID
ranks <- setNames(res_df$stat, res_df$gene)
ranks <- sort(ranks, decreasing = TRUE)
Check length(ranks) before continuing. A whole-blood RNA-seq run at 30 million reads typically leaves 13,000 to 16,000 genes after the count filter. If you have 45,000, your filter did not run and you are about to test thousands of noise genes.
3. Choose gene sets and, for ORA, the right background
Gene set choice determines what you can possibly find. MSigDB Hallmark is 50 sets of about 200 genes each, curated to be non-redundant, and it is the right default for a first pass. Gene Ontology Biological Process has more than 15,000 terms with heavy parent-child overlap, which gives resolution at the cost of a results table full of near-duplicates. Reactome sits between the two.
library(msigdbr)
h <- msigdbr(species = "Homo sapiens", collection = "H") # older versions: category = "H"
sets_h <- split(h$ensembl_gene, h$gs_name)
For ORA the background, called the universe, must be the set of genes that could have been called significant. That is the genes remaining after your count filter. The universe does not include every gene in the genome. Using a genome-wide universe for a whole-blood experiment produces a confident-looking list of immune terms that reflects only which genes are expressed in blood. This is the single most common way enrichment output becomes uninterpretable.
4. Run GSEA with fgsea
fgseaMultilevel estimates low p-values by adaptive multilevel splitting rather than plain permutation, so it resolves p-values below the 1/nperm floor that limits the classic implementation.
library(fgsea)
set.seed(42)
fg <- fgseaMultilevel(
pathways = sets_h,
stats = ranks,
minSize = 15,
maxSize = 500,
eps = 0
)
fg <- fg[order(fg$padj), ]
Read three columns. NES is the normalized enrichment score, positive when the set is concentrated among up-ranked genes. padj is the Benjamini-Hochberg adjusted p-value across the sets you tested. leadingEdge is the subset of genes that carry the signal, and it is the part you will spend time on.
Set minSize = 15 deliberately. Sets smaller than that produce unstable enrichment scores and inflated NES values. Sets larger than 500 are usually too vague to interpret. Treat padj < 0.05 as the reporting threshold and look at the whole ranked table anyway, since a set at padj = 0.08 with a coherent leading edge is worth as much as one at 0.04.
5. Cross-check with over-representation analysis
Now run the other family, with an explicit universe, on genes that passed your significance and effect size filters.
library(clusterProfiler)
library(org.Hs.eg.db)
sig <- res_df$gene[!is.na(res_df$padj) & res_df$padj < 0.05 &
abs(res_df$log2FoldChange) > 0.5]
ego <- enrichGO(
gene = sig,
universe = res_df$gene, # all tested genes
OrgDb = org.Hs.eg.db,
keyType = "ENSEMBL",
ont = "BP",
pAdjustMethod = "BH",
pvalueCutoff = 0.05,
qvalueCutoff = 0.2,
readable = TRUE
)
If sig has fewer than about 30 genes, ORA has almost no power and will return nothing. That is an accurate result. It does not mean your code has a bug. Report the GSEA table and say that the gene-level signal was too sparse for ORA.
6. Collapse redundancy before you read anything
Raw enrichment output overstates how many independent findings you have. Twelve GO terms sharing 80 percent of their genes are one finding. Collapse first, interpret second.
main <- collapsePathways(fg[fg$padj < 0.05, ], sets_h, ranks)
fg_main <- fg[fg$pathway %in% main$mainPathways, ]
ego_s <- simplify(ego, cutoff = 0.7, by = "p.adjust", measure = "Wang")
Then open the leading edge for each surviving set and look at the genes by name.
le <- fg_main$leadingEdge[[1]]
res_df[res_df$gene %in% le, c("gene", "log2FoldChange", "padj")]
A set driven by fifteen genes spread across the pathway is a different object from one driven by three neighbors on the same chromosome or by a single haemoglobin gene. The second case usually indicates a technical artifact such as globin carryover or a batch of samples processed together. Plot the enrichment curve with plotEnrichment(sets_h[[name]], ranks) and check that the running score rises steadily instead of jumping once.
7. Score pathways per sample so you can track them over time
Cohort methods answer questions about groups. With a longitudinal profile of one person, the more useful object is a pathway score per sample, which turns 15,000 noisy gene trajectories into 50 interpretable ones. GSVA computes this without reference to any contrast.
library(GSVA)
vsd <- vst(dds, blind = FALSE)
par <- gsvaParam(
exprData = assay(vsd),
geneSets = sets_h,
kcdf = "Gaussian",
minSize = 15,
maxSize = 500
)
es <- gsva(par) # 50 x n_samples matrix
Use kcdf = "Gaussian" for variance-stabilized or log-CPM input and kcdf = "Poisson" for raw integer counts. The scores are relative within the matrix you supply, so adding samples later changes earlier values. Recompute the full matrix each time rather than appending. This per-sample framing is the same move that single-gene multi-omics studies use when they relate one gene to immune and metabolic programs across a cohort 34.
Then model the score against time or against another measurement you hold, such as mean overnight glucose from a continuous monitor. Keep the number of tested pathways small and fixed in advance, because testing 15,000 GO terms against a glucose covariate across twelve timepoints will produce hits by construction.
8. Handle proteomics and variants differently
Proteomics enrichment follows the same code path with one change that governs the whole result. The background is the panel, never the genome. An affinity panel of 3,000 proteins is deliberately weighted toward inflammation, cardiometabolic, and oncology targets. Testing its significant hits against a genome-wide universe returns immune terms with tiny p-values that mean nothing. Set universe to every protein the assay attempted, including those that failed detection in this sample. Mass spectrometry has the same problem in a different shape, since detection depends on abundance and peptide properties.
prot_universe <- rownames(panel_annotation) # all assayed, not all detected
ego_p <- enrichGO(gene = sig_proteins, universe = prot_universe,
OrgDb = org.Hs.eg.db, keyType = "SYMBOL", ont = "BP")
Germline variants from your genome are a harder case, and our view is that pathway enrichment on a single person’s variant list is not informative. Everyone carries rare missense variants in thousands of genes. Longer genes accumulate more of them, and mutation rate varies across the genome, so any gene set containing long genes will look enriched. Rare variant burden tests work by comparing many genomes against many controls, which you do not have with n of 1. Use your variant data for annotation of specific genes that appear in a leading edge, and take any clinically actionable variant to a genetics clinician rather than interpreting it from an enrichment table.
Integration across layers is where enrichment earns its place. When a pathway appears in the RNA ranking and the same pathway’s protein products move in the same direction, the evidence is far stronger than either layer alone. Multi-omics studies use exactly this convergence to narrow candidate lists before validation 56. Treat every enrichment result as a hypothesis that needs an independent measurement, which is how the published workflows use it 78.
Common problems
Identifier mismatch produces silent failures rather than errors. Check the overlap before you trust any output: length(intersect(names(ranks), unique(unlist(sets_h)))) should cover most of your expressed genes. If it is near zero, you are joining Ensembl IDs against symbols or against Entrez IDs.
Ties in the ranking vector break the enrichment score. If your differential expression tool floors p-values at zero or you ranked by -log10(p) without a sign, large blocks of genes share a value and fgsea will warn about ties. Rank by the signed test statistic and confirm with sum(duplicated(ranks)) / length(ranks) that ties are a small fraction.
Too few replicates is the usual limit in personal data. DESeq2 will run with two samples per group and its dispersion estimates will be poor. With repeated sampling of one person, treat timepoints within a condition as replicates and block on draw date. Consecutive draws are correlated, so a paired design or limma with duplicateCorrelation is more defensible than treating each timepoint as independent.
Circularity is easy to introduce and hard to see. Selecting genes by a contrast and then testing gene sets built from that same contrast guarantees enrichment. Use gene sets from an external database and keep any sets you derived yourself out of the primary analysis.
KEGG deserves a note. The enrichKEGG function queries a live API, so results change over time and the runs are not reproducible without caching. Download a dated snapshot and store it with your analysis if you use KEGG at all.
Finally, direction and mechanism are not the same thing. A negative NES on an oxidative phosphorylation set tells you that the genes annotated to that set sit low in your ranking on that day. The score does not tell you that mitochondrial function changed. The score also does not support any decision about supplements, diet, or medication. Bring findings that seem clinically relevant to a physician who can order confirmatory testing.
Footnotes
-
Tianzhi Li, Xiaojia Kang, Sijie Zhang, et al. Integrating machine learning and multi-omics analysis to reveal nucleotide metabolism-related immune genes and their functional validation in ischemic stroke. Frontiers in Immunology, 2025. https://doi.org/10.3389/fimmu.2025.1561544 ↩
-
Jiacheng Wu, Yulu Yang, Jianwu Huang, et al. Identification of key biomarkers for myocardial infarction by multi-omics analysis and machine learning. Frontiers in Immunology, 2026. https://doi.org/10.3389/fimmu.2026.1711521 ↩
-
Wencheng Gong, Minghui Ge, Xiaotong Xi, et al. ALAD as a prognostic biomarker regulates metabolism and immune responses in renal cell carcinoma through multi-omics analysis. Translational Oncology, 2025. https://doi.org/10.1016/j.tranon.2025.102524 ↩
-
Yunzheng Zhao, Qingyu Li, Jiajun Li, et al. Expression and clinical significance of FANCI gene in pan-cancer: a comprehensive analysis based on multi-omics data. Frontiers in Genetics, 2025. https://doi.org/10.3389/fgene.2025.1542888 ↩
-
Xu-Sheng Liu, Xi Chen, Zhi-Ming Cheng, et al. Integrated multi-omics and functional analysis uncovers the structural and regulatory significance of Nucleophosmin 1 in the metabolic progression of esophageal carcinoma. International Journal of Biological Macromolecules, 2026. https://doi.org/10.1016/j.ijbiomac.2025.149610 ↩
-
Valentin Costes, Eli Sellem, Sylvain Marthey, et al. Multi-omics data integration for the identification of biomarkers for bull fertility. PLOS ONE, 2024. https://doi.org/10.1371/journal.pone.0298623 ↩
-
Jirong Jie, Qiong Wang, Zhifeng Chen, et al. Multi-omics analysis and preliminary experimental validation of acetyl tributyl citrate (ATBC) promoting bladder cancer progression via the AKR1B1/EMT axis. Environmental Pollution, 2026. https://doi.org/10.1016/j.envpol.2026.127970 ↩
-
Yuanyuan Li, Limin Nie, Tianli Lv, et al. Investigation into the molecular mechanism of obesity: an integrated approach of multi-omics analysis, machine learning and experimental validation. Journal of Translational Medicine, 2025. https://doi.org/10.1186/s12967-025-07096-9 ↩
Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.