How to Run a Differential Gene Expression Analysis on Your Own RNA-Seq Data
By the end of this guide you will have a gene-level count matrix, a fitted statistical model, and a ranked table of genes with log2 fold changes, standard errors, p-values, and false discovery rates. You need paired-end FASTQ files from at least two samples you want to compare. You also need a reference transcriptome and genome FASTA, a GTF annotation, and a machine with about 16 GB of RAM and 100 GB of free disk. The software is fastp, salmon, R 4.3 or later, and the Bioconductor packages tximport, DESeq2, edgeR, and limma. If your samples are repeated draws from a single person over time, read step 6 before you run anything, because the design matrix is where that case is won or lost.
1. Lay out the data and write the sample sheet first
The single most common source of wasted compute is a mislabeled sample. Write the sample sheet before you touch the reads, and treat it as the source of truth for every downstream step.
Create a flat directory of FASTQ files and a samples.tsv with one row per library. Include the columns you intend to model and nothing else you will be tempted to fish in.
sample subject timepoint condition batch fq1 fq2
S01 P1 T0 fasted B1 raw/S01_R1.fastq.gz raw/S01_R2.fastq.gz
S02 P1 T1 fed B1 raw/S02_R1.fastq.gz raw/S02_R2.fastq.gz
S03 P1 T0 fasted B2 raw/S03_R1.fastq.gz raw/S03_R2.fastq.gz
Record library prep date and blood collection tube type if you have them. Also record RIN, the RNA integrity number, which is a 1 to 10 score of RNA degradation. These become covariates or explanations later. Field-standard best-practice reviews put sample metadata and experimental design ahead of tool choice for exactly this reason.1
2. Trim and QC the reads
Adapter read-through and low-quality tails inflate the fraction of unmapped reads and can bias short-transcript quantification. We use fastp because it does adapter detection, quality trimming, and reporting in one pass at high throughput.
fastp \
-i raw/S01_R1.fastq.gz -I raw/S01_R2.fastq.gz \
-o trim/S01_R1.fq.gz -O trim/S01_R2.fq.gz \
--detect_adapter_for_pe \
--cut_tail --cut_tail_mean_quality 20 \
--length_required 36 \
--json qc/S01.fastp.json --html qc/S01.fastp.html \
--thread 8
Then aggregate every JSON with multiqc qc/ and look at three numbers per sample: duplication rate, GC distribution shape, and the fraction of reads surviving. A bimodal GC curve in a whole-blood library usually means globin or ribosomal carryover rather than contamination.
Do not trim aggressively for quantification. Removing bases below Q20 across the whole read costs you sensitivity and buys almost nothing, because modern quantifiers handle mismatches probabilistically.
3. Quantify with salmon and a decoy-aware index
You have two reasonable choices. Alignment-free quantification with salmon or kallisto gives transcript-level estimates in minutes per sample. Spliced alignment with STAR followed by featureCounts gives you a BAM file you can look at, which matters if you also want variant calls or splicing analysis from the same reads.2
We default to salmon for differential expression and add a STAR run only when a gene looks surprising and we want to see the pileup. Build the index against a gentrome, which is the transcriptome concatenated with the genome used as decoy sequence. Without decoys, reads from unannotated genomic regions get forced onto real transcripts and inflate their counts.
# GENCODE human release, transcripts + primary assembly
grep "^>" <(gunzip -c GRCh38.primary_assembly.genome.fa.gz) | cut -d " " -f 1 \
| sed 's/>//g' > decoys.txt
cat gencode.v45.transcripts.fa.gz GRCh38.primary_assembly.genome.fa.gz > gentrome.fa.gz
salmon index -t gentrome.fa.gz -d decoys.txt -p 16 -i salmon_idx_v45 --gencode
Then quantify each sample. The bias flags matter more than people expect for libraries prepared on different days.
salmon quant -i salmon_idx_v45 -l A \
-1 trim/S01_R1.fq.gz -2 trim/S01_R2.fq.gz \
--validateMappings --seqBias --gcBias --posBias \
--numBootstraps 30 \
-p 16 -o quant/S01
Check quant/S01/logs/salmon_quant.log for the mapping rate. For a polyA-selected human library expect 75 to 92 percent. Below 60 percent, stop and find out why before proceeding.
4. Import to gene level with tximport
Transcript-level estimates are the natural output of salmon, but gene-level differential expression is better powered and far easier to interpret. tximport with countsFromAbundance = "lengthScaledTPM" aggregates transcripts to genes while correcting for changes in average transcript length across samples.
library(tximport); library(readr)
samples <- read_tsv("samples.tsv")
files <- file.path("quant", samples$sample, "quant.sf")
names(files) <- samples$sample
# tx2gene: two columns, TXNAME and GENEID, from the GTF
tx2gene <- read_tsv("tx2gene.v45.tsv")
txi <- tximport(files, type = "salmon", tx2gene = tx2gene,
countsFromAbundance = "lengthScaledTPM",
ignoreTxVersion = FALSE)
dim(txi$counts) # ~62,000 genes x n samples
If you use countsFromAbundance = "no" instead, you must pass the offsets to DESeq2 via DESeqDataSetFromTximport, which handles length correction internally. Both are correct. Mixing them is not.
5. Filter low-count genes and normalize
Of roughly 62,000 annotated GENCODE genes, only 12,000 to 16,000 are meaningfully expressed in any one tissue. Testing the rest costs you multiple-testing power and produces unstable fold changes. Filter with edgeR::filterByExpr, which sets the count threshold from library size and the smallest group size rather than from an arbitrary CPM cutoff.
library(edgeR); library(DESeq2)
dds <- DESeqDataSetFromTximport(txi, colData = samples,
design = ~ condition)
keep <- filterByExpr(counts(dds), group = samples$condition)
dds <- dds[keep, ]
Normalization corrects for differences in sequencing depth and, more importantly, for composition. If a handful of genes soak up most of the reads in one condition, raw depth scaling makes every other gene look down-regulated. TMM (trimmed mean of M-values) estimates a scaling factor from the trimmed log-ratio distribution and was introduced precisely to handle this failure mode.3 DESeq2’s median-of-ratios estimator solves the same problem by a different route, and in practice the two agree closely.
This matters concretely in whole blood, where hemoglobin transcripts can consume 30 to 70 percent of reads without globin depletion. Check it before trusting anything:
cpm_mat <- cpm(counts(dds))
globin <- c("HBB","HBA1","HBA2","HBD")
colSums(cpm_mat[rownames(cpm_mat) %in% globin, , drop = FALSE]) / 1e4 # percent
6. Choose the model, and get the design matrix right
Three tools dominate and all three are defensible. DESeq2 and edgeR both fit negative binomial generalized linear models with empirical Bayes shrinkage of the per-gene dispersion. limma-voom transforms counts to log-CPM with precision weights and then uses the linear model machinery from microarray analysis. Reviews of differential expression pipelines consistently find these three near the top on real data.4
Our default is DESeq2 for small designs (fewer than about eight samples) because its dispersion shrinkage is aggressive and it handles low replication gracefully. We switch to limma-voom when the design has repeated measures on the same individual, because duplicateCorrelation gives a clean way to model within-subject correlation.
For a simple two-group comparison:
design(dds) <- ~ batch + condition
dds <- DESeq(dds)
res <- results(dds, contrast = c("condition","fed","fasted"), alpha = 0.05)
For repeated sampling of the same person across conditions, put the subject in the model as a blocking factor. This is the paired design and it is a large power gain, because it removes between-person variation from the error term.
design(dds) <- ~ subject + condition # subject-specific intercepts
If every sample comes from one person and you have several timepoints per condition, the same syntax applies with timepoint replacing subject only if timepoints are nested correctly. With a single subject and no replication within a condition, no tool can separate biological signal from technical noise. In that case treat fold changes as descriptive and do not report p-values. Human gene expression varies substantially between individuals and across time within an individual, and that variation is the thing an unreplicated comparison confounds with your effect.5
The limma-voom route with within-subject correlation:
library(limma)
dge <- DGEList(counts(dds)); dge <- calcNormFactors(dge, method = "TMM")
mm <- model.matrix(~ batch + condition, data = samples)
v <- voom(dge, mm, plot = TRUE)
cor <- duplicateCorrelation(v, mm, block = samples$subject)
v <- voom(dge, mm, block = samples$subject,
correlation = cor$consensus, plot = FALSE)
fit <- lmFit(v, mm, block = samples$subject, correlation = cor$consensus)
fit <- eBayes(fit)
topTable(fit, coef = "conditionfed", number = 50, sort.by = "P")
7. Shrink the fold changes and set a threshold you can defend
Raw maximum-likelihood log2 fold changes for low-count genes are enormous and meaningless. Apply apeglm shrinkage, which pulls poorly estimated effects toward zero while leaving well-estimated large effects alone.
library(apeglm)
resLFC <- lfcShrink(dds, coef = "condition_fed_vs_fasted", type = "apeglm")
Then test against a fold-change threshold rather than filtering on it after the fact. Filtering a significance list by abs(log2FC) > 1 breaks the FDR guarantee. Testing directly does not.
res_thr <- results(dds, contrast = c("condition","fed","fasted"),
lfcThreshold = 0.585, altHypothesis = "greaterAbs")
sum(res_thr$padj < 0.05, na.rm = TRUE)
lfcThreshold = 0.585 corresponds to a 1.5-fold change. We prefer this over a 2-fold cutoff for tissue-level bulk RNA, where true regulatory effects are often in the 20 to 60 percent range.
8. Sanity-check the result before you interpret it
Three diagnostics catch most disasters. First, plot a PCA on variance-stabilized counts and see whether PC1 tracks your condition or your batch.
vsd <- vst(dds, blind = FALSE)
plotPCA(vsd, intgroup = c("condition","batch"))
Second, look at the p-value histogram. Under the null it should be flat with a spike near zero. A U-shaped or hill-shaped histogram means the model is misspecified, usually because a real covariate is missing.
Third, plot dispersion estimates with plotDispEsts(dds). The fitted trend should decrease monotonically with mean expression, and the shrunk per-gene estimates should cluster around it. Scattered points far above the trend with no shrinkage indicate dispersion outliers, which DESeq2 flags and handles but you should still look at.
In whole blood, also check cell composition. Neutrophil fraction routinely dominates the first principal component of blood transcriptomes and will masquerade as a condition effect if your sampling times differ. Estimate proportions with a deconvolution method and add the dominant fraction as a covariate.
9. Interpret with restraint
You now have a ranked gene list. Enrichment analysis on that list (clusterProfiler::gseGO or fgsea against MSigDB Hallmark) gives pathway-level summaries that are more stable than any individual gene. Use the full ranked statistic rather than a cut list, because threshold-free GSEA is less sensitive to your arbitrary cutoff.
Two limits are worth stating plainly. Differential expression tools were developed and tuned on protein-coding mRNA, and their performance degrades markedly on long non-coding RNA, where counts are lower and more variable.6 Treat lncRNA hits as hypotheses requiring targeted follow-up. Separately, gene-level differential expression will miss isoform switches where total output is unchanged but which transcript is made changes. That question needs transcript-level testing, an analysis with its own history and its own tradeoffs.7
Nothing in this output is a clinical result. Expression changes in a blood transcriptome reflect cell composition and time of day at least as much as anything durable. They also reflect recent meals, sleep, and infection status. If a finding concerns you or you want it acted on, take it to a physician who can order validated clinical testing.
Common problems
Mapping rate below 60 percent. Usually a wrong or mismatched index, or rRNA carryover from a failed depletion. Run the reads against a small rRNA reference to check, and confirm your GTF release matches the FASTA release exactly.
Zero significant genes. If the PCA shows no separation by condition, the effect is smaller than your noise and more samples are the only fix. If PCA separates cleanly but nothing passes FDR, check that your design includes the blocking factor and that you have not accidentally fit ~ condition when subjects are paired.
Thousands of significant genes with tiny fold changes. This is typically a composition artifact or an unmodeled batch. Verify normalization factors span less than roughly a 3-fold range with sizeFactors(dds), and inspect globin and mitochondrial gene fractions.
A U-shaped p-value histogram. Add the missing covariate. Candidates are batch, RIN, sex, and estimated cell-type proportions.
One sample far from all others on PCA. Compute cooks.distance from the DESeq2 fit and check whether that sample drives many of your hits. Removing an outlier is legitimate if you decide on the criterion before looking at the results and you report the removal.
Results change when you switch tools. Some disagreement between DESeq2, edgeR, and limma-voom is expected at the margins. It concentrates in low-count genes.2 If the top 100 genes disagree substantially, the design or the normalization is the problem. The tool is not at fault.
Comparing your sample to a public cohort. Pulling GTEx or a published dataset as a control group mostly measures the batch difference between two labs. Reference data is useful for orienting yourself on absolute expression levels. It is not a valid control arm for a differential test.
Footnotes
-
Ana Conesa, Pedro Madrigal, Sonia Tarazona, et al. A survey of best practices for RNA-seq data analysis. Genome Biology, 2016. https://doi.org/10.1186/s13059-016-0881-8 ↩
-
S. Akila Parvathy Dharshini, Y.-H. Taguchi, M. Michael Gromiha. Identifying suitable tools for variant detection and differential gene expression using RNA-seq data. Genomics, 2020. https://doi.org/10.1016/j.ygeno.2019.12.011 ↩ ↩2
-
Mark D Robinson, Alicia Oshlack. A scaling normalization method for differential expression analysis of RNA-seq data. Genome Biology, 2010. https://doi.org/10.1186/gb-2010-11-3-r25 ↩
-
Diletta Rosati, Maria Palmieri, Giulia Brunelli, et al. Differential gene expression analysis pipelines and bioinformatic tools for the identification of specific biomarkers: A review. Computational and Structural Biotechnology Journal, 2024. https://doi.org/10.1016/j.csbj.2024.02.018 ↩
-
Joseph K. Pickrell, John C. Marioni, Athma A. Pai, et al. Understanding mechanisms underlying human gene expression variation with RNA sequencing. Nature, 2010. https://doi.org/10.1038/nature08872 ↩
-
Alemu Takele Assefa, Katrijn De Paepe, Celine Everaert, et al. Differential gene expression analysis tools exhibit substandard performance for long non-coding RNA-sequencing data. Genome Biology, 2018. https://doi.org/10.1186/s13059-018-1466-5 ↩
-
Cole Trapnell, Adam Roberts, Loyal Goff, et al. Differential gene and transcript expression analysis of RNA-seq experiments with TopHat and Cufflinks. Nature Protocols, 2012. https://doi.org/10.1038/nprot.2012.016 ↩
Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.