Skip to content

How to run fgsea on your own RNA-seq data

Oak
A laboratory instrument sorting glowing filaments into a ranked vertical ladder, with bright bundles arcing into branching channels.

fgsea is an R package that runs preranked gene set enrichment analysis fast enough to estimate very small p-values accurately. You give it two things: a named numeric vector of genes ranked by some statistic, and a list of gene sets. For each set it returns an enrichment score, a normalized enrichment score, and a p-value. It also returns the subset of genes driving the signal. If you have differential expression results from DESeq2 or limma and you want to know which biological processes moved, fgsea is the tool we would reach for first.

The reason to prefer it over the original Broad GSEA implementation is the p-value estimation. Classic GSEA permutes gene labels a fixed number of times, so the smallest p-value you can report is bounded by the permutation count. fgsea’s multilevel algorithm uses adaptive sampling to estimate arbitrarily small p-values, which matters once you correct for testing several thousand gene sets.

Building the ranked vector

Nearly every bad fgsea result traces back to the ranking vector rather than the algorithm. The statistic you rank by defines what “enriched” means, so choose it deliberately.

For DESeq2 output, we rank by the Wald statistic in the stat column. It combines effect size and precision, so a gene with a large fold change and huge dispersion does not float to the top. Ranking by raw log2FoldChange gives lowly expressed genes with wild ratios undue influence. Ranking by -log10(pvalue) alone throws away direction and breaks the whole test. If you want a signed p-value ranking, use sign(log2FoldChange) * -log10(pvalue) and accept that ties pile up at the extremes.

Three mechanical problems bite here. First, stat is NA for genes DESeq2 filtered out by independent filtering, so drop those rows before building the vector. Second, mapping Ensembl IDs to HGNC symbols is many-to-one, and duplicate symbols must be collapsed. We keep the entry with the largest absolute statistic rather than averaging, because averaging a strong positive and a strong negative produces a meaningless zero. Third, fgsea will warn about ties in the preranked list. A handful of ties is fine. A vector where hundreds of genes share the same value means your statistic is saturating and the enrichment score walk is no longer well defined.

library(fgsea); library(data.table); library(msigdbr)

res <- as.data.table(DESeq2::results(dds, name = "condition_B_vs_A"),
                     keep.rownames = "ensembl")
res <- res[!is.na(stat)]
res[, symbol := mapIds(org.Hs.eg.db, ensembl, "SYMBOL", "ENSEMBL")]
res <- res[!is.na(symbol)]
res <- res[order(-abs(stat))][!duplicated(symbol)]
ranks <- setNames(res$stat, res$symbol)

Choosing gene sets

The gene sets matter as much as the ranking. msigdbr pulls MSigDB collections directly into R as a data frame, which avoids parsing GMT files and keeps species mapping explicit.

We start with Hallmark (collection = "H"), fifty sets of roughly 200 genes each, curated to reduce the redundancy that makes GO output so hard to read. For a second pass we add Reactome (C2, subcollection CP:REACTOME) and GO Biological Process (C5, GO:BP). Running all of C5 at once gives you several thousand overlapping sets and a results table that is mostly the same signal restated at different levels of a hierarchy.

Size filtering is not cosmetic. minSize = 15 removes sets too small to produce a stable enrichment score. maxSize = 500 removes sets so broad that almost any ranking hits them. Both are applied after intersecting the set with the genes present in your ranking, which is why a set listed as having 300 genes may be reported with size = 180.

Parameters we use

The modern entry point is fgsea(), which dispatches to the multilevel algorithm. The call we run by default looks like this.

pathways <- split(msigdbr(species = "Homo sapiens", collection = "H")$gene_symbol,
                  msigdbr(species = "Homo sapiens", collection = "H")$gs_name)

fgseaRes <- fgsea(pathways = pathways,
                  stats    = ranks,
                  minSize  = 15,
                  maxSize  = 500,
                  eps      = 0,
                  nproc    = 4)

Setting eps = 0 removes the lower bound on reported p-values. The default is nonzero, so extremely enriched pathways get clamped and report a boundary value instead of their true p-value. Setting it to zero costs runtime but gives you numbers you can rank. nPermSimple controls the initial sampling stage and only needs raising if fgsea warns that the estimate is unstable. Use scoreType = "pos" when your statistic has no meaningful sign, for example a vector of absolute correlation values.

Reading the output

The results table has one row per gene set. ES is the raw enrichment score, the maximum deviation of the running-sum walk down the ranked list. NES normalizes that score by set size so values are comparable across sets, and it is the column to sort by. padj is a Benjamini-Hochberg adjustment across all tested sets. leadingEdge is a list column holding the genes that appear before the running sum peaks.

The leading edge is the part worth reading closely. A significant NES tells you a set moved. The leading edge tells you which forty genes did the moving, and whether they are a coherent module or a scattering of loosely related transcripts. We export it as a collapsed string before writing to CSV, because fwrite will not serialize a list column.

Redundancy is the next problem. Reactome and GO will hand you twenty significant sets that share most of their leading-edge genes. collapsePathways(fgseaRes[padj < 0.05], pathways, ranks) re-runs enrichment conditioned on the top set and returns a parent list. That cuts a table of 200 hits down to something you can read.

Where fgsea gives you confident nonsense

Two failure modes recur. The first is a ranking built from an underpowered contrast. fgsea will happily rank noise and return significant sets, because the permutation null is over gene positions rather than over samples. Gene-level permutation does not account for correlation between genes, so a ranking driven by a single sample outlier can produce small p-values with no replication behind it.

The second is applying fgsea to a single person’s single sample. There is no contrast, so there is no ranking. Comparing one person’s expression to a reference cohort mean is a pairwise design with n=1 on one side, and pairwise approaches are less stable than single-sample scoring. A systematic evaluation of GSEA tools found single-sample methods gave more reproducible biological signal than pairwise comparison across transcriptomic datasets 1. The dualGSEA approach runs both and reports where they agree, which is a reasonable way to see how much of your result depends on the method 2.

For one person, the design that works is longitudinal. Sequence the same tissue at several time points, treat time as the contrast, and use the person as their own control. That removes between-individual variation, which is the dominant term in most expression comparisons. Multi-timepoint designs analyzed this way have resolved temporal and sex-specific metabolic programs that cross-sectional comparison misses 3.

Extending past RNA

If you hold proteomics and metabolomics alongside transcriptomics, running fgsea separately per layer and eyeballing the overlap is weak practice. multiGSEA maps features from transcript, protein, and metabolite layers onto shared pathway definitions and aggregates p-values across layers before correction 4. mitch handles the related problem of multiple contrasts, scoring pathway enrichment jointly across profiles rather than one at a time 5. Both build on the same rank-based machinery. A survey of multi-omics pathway analysis methods lays out where the integration assumptions differ and which ones carry real statistical cost 6.

None of this output is a clinical result. An enriched inflammatory pathway in your blood transcriptome is a measurement about gene sets. It is not a finding about your health. Take anything that looks concerning to a physician who can order confirmatory testing.

Questions people also ask

What is the difference between fgsea and the original GSEA?

Both compute the same Kolmogorov-Smirnov-style enrichment statistic on a preranked list. The difference is how the null distribution is estimated. fgsea uses a multilevel adaptive sampling scheme that reaches very small p-values without running millions of permutations, so a full MSigDB run finishes in seconds rather than hours.

Should I use fgseaSimple or fgseaMultilevel?

Use fgsea(), which calls the multilevel algorithm. fgseaSimple exists for backward compatibility and bounds your p-values at roughly 1/nperm, which becomes limiting after multiple testing correction across thousands of sets.

How many genes do I need in my ranking?

The full expressed transcriptome, typically 12,000 to 18,000 genes after filtering. Do not pre-filter to significant genes. The method depends on the position of set members across the entire ranking, and truncating the list destroys the null.

Can I run fgsea on proteomics data?

Yes. Rank proteins by the moderated t-statistic from limma and map UniProt accessions to gene symbols. Coverage is the limit: a typical plasma proteomics panel measures far fewer analytes than an RNA-seq run, so many gene sets will fall below minSize after intersection.

What does a negative NES mean?

The set is enriched at the bottom of your ranked list, meaning its genes are coordinately lower in the numerator condition of your contrast. Direction follows entirely from how you built the ranking, so record the contrast direction alongside the results.

Footnotes

  1. Courtney Bull, Ryan M Byrne, Natalie C Fisher, et al. Evaluation of Gene Set Enrichment Analysis (GSEA) tools highlights the value of single sample approaches over pairwise for robust biological discovery. 2024. https://doi.org/10.1101/2024.03.15.585228 ↩

  2. Courtney Bull, Ryan M. Byrne, Natalie C. Fisher, et al. Dual gene set enrichment analysis (dualGSEA); an R function that enables more robust biological discovery and pre-clinical model alignment from transcriptomics data. Scientific Reports, 2024. https://doi.org/10.1038/s41598-024-80534-8 ↩

  3. Yuan Gao, Zhangyi Tu, Qi Zhao, et al. Hepatic metabolic adaptation to endurance exercise: temporal and sex differences by multiomics integration and validation. Cell Communication and Signaling, 2026. https://doi.org/10.1186/s12964-026-03154-x ↩

  4. Sebastian Canzler, Jörg Hackermüller. multiGSEA: a GSEA-based pathway enrichment analysis for multi-omics data. BMC Bioinformatics, 2020. https://doi.org/10.1186/s12859-020-03910-x ↩

  5. Antony Kaspi, Mark Ziemann. mitch: multi-contrast pathway enrichment for multi-omics and single-cell profiling data. BMC Genomics, 2020. https://doi.org/10.1186/s12864-020-06856-9 ↩

  6. Zeynab Maghsoudi, Ha Nguyen, Alireza Tavakkoli, et al. A comprehensive survey of the approaches for pathway analysis using multi-omics data integration. Briefings in Bioinformatics, 2022. https://doi.org/10.1093/bib/bbac435 ↩

Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.