Skip to content

DESeq2 baseMean Cutoff: Filter Counts Before the Test

Oak
A coral-like filtering weir spans a glowing river at dusk, sieving drifting spores, with a collapsed second weir downstream.

The short answer is that you should not apply a baseMean cutoff to a DESeq2 results table. Filter on raw counts before you call DESeq(), using a rule like “at least 10 counts in at least as many samples as the smallest group,” and then let DESeq2’s built-in independent filtering handle the remaining low-information genes. If you do decide to drop low-abundance genes after the fact, you have to recompute the Benjamini-Hochberg adjustment on the subset, because removing rows changes every adjusted p-value in the table. A cutoff applied to res$padj without that recomputation gives you a number that no longer means what the column header says.

What baseMean is

baseMean is the mean of the normalized counts for a gene across all samples in the dataset, pooled over every condition. Normalized here means raw counts divided by that sample’s size factor, the median-of-ratios scaling that DESeq2 computes to put libraries on a comparable footing. It is not a TPM, not an FPKM, and not corrected for transcript length.

That last point matters more than people expect. A 400 bp transcript and a 12 kb transcript at identical molar concentration will have very different baseMean values, because longer transcripts generate more fragments. So baseMean is a good within-gene, across-sample quantity and a poor across-gene one. If someone tells you gene A is expressed 250 times more than gene B because baseMean is 20000 versus 80, they have made a length-confounded comparison.

What baseMean is genuinely good for is judging how much information the model had. A gene with baseMean 3 in a 12-sample experiment was seen a few dozen times total. Its log2 fold change estimate is dominated by Poisson noise, and that is the real concern people are trying to address when they reach for a cutoff.

Why a post hoc baseMean cutoff is the wrong tool

There are two problems with filtering the results table. The first is statistical. The BH procedure controls the false discovery rate over a fixed set of tests. Deleting rows after the adjustment leaves you with p-adjusted values computed against a denominator that no longer exists. You will usually end up conservative, but you cannot say by how much, and the FDR you report is not the FDR you controlled.

The second problem is that the cutoff is redundant. DESeq2 already does a principled version of this inside results(). Independent filtering uses baseMean as a filter statistic, which is valid precisely because the mean of normalized counts is (approximately) independent of the test statistic under the null. It sweeps candidate quantile thresholds, applies BH to each surviving subset, and keeps the threshold that maximizes the number of genes below your target alpha.

You can see exactly what it chose:

res <- results(dds, alpha = 0.05)
metadata(res)$filterThreshold      # baseMean value at the chosen quantile
metadata(res)$alpha
plot(metadata(res)$filterNumRej, type = "b",
     xlab = "quantiles of baseMean", ylab = "number of rejections")

Genes below that threshold come back with padj = NA. They do not come back with a small p-value you then have to argue about. Genes flagged as count outliers by Cook’s distance also get NA. Those NAs are the machinery working. They are not a bug to patch over with subset(res, baseMean > 50).

The pre-filter we use

Do the filtering once, on the count matrix, before fitting. This is the step that helps, because it reduces the number of tests and improves dispersion estimation by removing rows that carry no signal.

smallestGroupSize <- min(table(colData(dds)$condition))
keep <- rowSums(counts(dds) >= 10) >= smallestGroupSize
dds  <- dds[keep, ]
dds  <- DESeq(dds)

The >= 10 threshold is on raw counts, and the rowSums form is deliberate: it keeps a gene that is off in one group and clearly on in the other, which a mean-based filter like rowMeans(counts) > 10 can discard. For a typical human polyA library at 30 to 50 million reads, starting from roughly 60,000 GENCODE gene entries, this rule leaves somewhere in the range of 14,000 to 20,000 genes. If it leaves 40,000, check your annotation and your rRNA and globin fraction. If it leaves 8,000, check library depth and duplication rate.

Two caveats on where this rule breaks. With very unbalanced designs, say 3 versus 30 samples, smallestGroupSize is 3 and the filter becomes permissive. Consider a fixed floor instead. And if your biology lives in low-abundance transcripts, a blunt count filter will remove it. Work on circular and linear non-coding RNA in aged human brain tissue depends on quantifying species that sit far below the abundance of protein-coding mRNA, and an aggressive pre-filter would delete the object of study. 1 Choose the filter with your target transcript class in mind.

When low baseMean is really the problem, shrink the fold change

Most people who want a baseMean cutoff have been burned by a low-count gene showing a log2 fold change of 6 with a plausible-looking adjusted p-value. The fix for that is shrinkage. Deletion is not the fix.

resLFC <- lfcShrink(dds, coef = "condition_treated_vs_control", type = "apeglm")

The apeglm estimator pulls fold changes toward zero in proportion to their uncertainty. A gene with baseMean 8 and huge standard error collapses toward zero. A gene with baseMean 5000 and a tight interval barely moves. The result is a ranking you can sort by effect size without a hidden abundance filter, and MA plots that stop being a fan of noise at the left edge.

If you want a biologically meaningful effect-size threshold as well, put it inside the test rather than applying it afterward:

res <- results(dds, lfcThreshold = 0.585, altHypothesis = "greaterAbs")

That tests the null that the absolute log2 fold change is below 0.585, which corresponds to a 1.5-fold change. It is a different and stricter question than the default null of zero, and it comes with correctly calibrated p-values. Post hoc filtering on abs(log2FoldChange) > 0.585 does not.

Checking your choices against something you can control

Filtering rules are easy to argue about and hard to settle with real data, because you never know the truth. Simulation is the way out, and it works only if the simulator reproduces the correlation structure of real expression data rather than drawing independent negative binomials per gene. 2 If a filter and shrinkage combination behaves sensibly on simulated data matched to your depth and sample size, you have some basis for the settings you used on your own samples.

For a personal profile with a handful of timepoints, the constraint is sample size. With n=2 or n=3 per condition, dispersion estimates lean heavily on the shared trend, and no filtering choice rescues that. Treat the output as a ranked hypothesis list. Any move from a ranked list to a statement about your health needs a clinician who can see the rest of your clinical picture.

Questions people also ask

What does baseMean mean in DESeq2? It is the average of the size-factor-normalized counts for that gene across every sample in the dataset, regardless of condition. It measures how much evidence the model had for that gene, and it is not length-normalized, so comparing baseMean between two different genes is not a valid abundance comparison.

What is the purpose of the DESeq2 statistical test? DESeq2 fits a negative binomial generalized linear model per gene, shares dispersion information across genes to stabilize variance estimates at small sample size, and tests whether a model coefficient (usually a log2 fold change between conditions) differs from a specified null. Independent filtering and BH adjustment then convert those per-gene p-values into an FDR-controlled gene list.

Is DESeq2 better than edgeR? They agree on most genes and disagree at the margins, so the choice depends on what you are optimizing. A recent systematic comparison across studies found the two tools differ measurably in sensitivity and in how well their results hold up across datasets, which argues against treating them as interchangeable defaults. 3 Our practice is to run DESeq2 with apeglm shrinkage for small designs and to check any result we plan to act on in edgeR-QLF as well.

What does a fold change of 1.5 mean? The gene’s expression estimate in one group is 1.5 times the other, which is a log2 fold change of about 0.585. A 1.5-fold change in a gene with baseMean 20,000 is usually well estimated. The same number in a gene with baseMean 15 is often noise, which is why you should look at the shrunken estimate and its interval rather than the raw ratio.

Footnotes

  1. Alexander Knyshov, Junming Hu, Mintao Lin, et al. Characterizing circular and linear non-coding RNA in human aged brains and their association with dementia. Scientific Reports, 2026. https://doi.org/10.1038/s41598-026-64189-1 ↩

  2. Jianing Yang, Gregory R. Grant, Thomas G. Brooks. Generating correlated data for omics simulation. PLOS Computational Biology, 2025. https://doi.org/10.1371/journal.pcbi.1013392 ↩

  3. Mostafa Rezapour. Tool choice matters: Evaluating edgeR vs. DESeq2 for sensitivity, robustness, and cross-study performance. PLOS One, 2026. https://doi.org/10.1371/journal.pone.0353788 ↩

Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.