Skip to content

Preprocessing Your Own FASTQ Files with fastp

Oak
A laboratory machine pulls a glowing ribbon through a narrow slot, trimming off dim frayed ends that fall into a tray.

By the end of this guide you will have trimmed and filtered FASTQ files for your own sequencing data, a machine-readable JSON quality report per sample, and a small table of metrics you can compare across runs and across years. You need the raw FASTQ files, gzipped and paired-end, typically *_R1_001.fastq.gz and *_R2_001.fastq.gz. You also need roughly the same amount of free disk again for outputs and a Linux or macOS machine with at least 8 cores. Everything here runs on a laptop for RNA-seq. A 30x whole genome at about 100 GB of compressed FASTQ wants a workstation or a cloud instance with fast local disk, because fastp is usually I/O bound rather than CPU bound. The tool to cite is the paper by Chen S, Zhou Y, Chen Y, and Gu J, titled “fastp: an ultra-fast all-in-one FASTQ preprocessor”. It appeared in Bioinformatics 2018, 34(17):i884–i890, with a 2023 update in iMeta describing the deduplication and one-pass design.

1. Install fastp and pin the version

Use conda or mamba if you already manage environments that way. The bioconda package is maintained and matches upstream releases closely.

mamba create -n qc -c conda-forge -c bioconda fastp=0.24.0
mamba activate qc
fastp --version   # fastp 0.24.0

If you do not want a package manager, the author ships a static Linux binary.

wget http://opengene.org/fastp/fastp.0.24.0
chmod a+x fastp.0.24.0
./fastp.0.24.0 --version

Building from source is worth it only if you want the libdeflate acceleration tuned for your machine. It needs libdeflate, libisal, and a C++11 compiler. The gain over the prebuilt binary is modest on most hardware.

Whichever route you choose, write the version into your notes now. Trimming parameters and tool versions are part of the data, not incidental detail, and reproducibility in omics workflows depends on recording them alongside the results1. If you re-sequence in two years and compare, you want to know whether a shift in duplication rate came from the sample or from a changed default.

2. Look at the reads before you touch them

Run one report-only pass with all modification disabled. This tells you what you have before any decision is made.

fastp -i sample_R1.fastq.gz -I sample_R2.fastq.gz \
  --disable_adapter_trimming \
  --disable_quality_filtering \
  --disable_length_filtering \
  --disable_trim_poly_g \
  --json qc/sample.raw.json --html qc/sample.raw.html \
  -w 4

Also read the first header line directly, because it names the instrument and the flow cell.

zcat sample_R1.fastq.gz | head -1
# @A00534:412:HVJK7DSXY:1:1101:2撒...

An instrument ID starting with A0 or LH indicates a NovaSeq or NovaSeq X, which uses two-color chemistry. On those platforms a base with no signal is called G, so read ends can accumulate long runs of high-quality G. fastp detects this from the header and enables poly-G trimming automatically, but you should know it is happening. From the raw report, note three numbers. They are summary.before_filtering.read1_mean_length, summary.before_filtering.q30_rate, and duplication.rate.

3. Trim whole-genome reads conservatively

Our recommendation for WGS destined for BWA-MEM or DRAGMAP is to remove adapters and platform artifacts and to leave base qualities alone. Modern aligners soft-clip adapter remnants and the variant caller uses base quality scores as evidence. Aggressive sliding-window trimming discards that evidence and shortens reads in a position-dependent way, which can subtly bias allele balance at heterozygous sites.

mkdir -p trim qc logs
fastp \
  -i raw/sample_R1.fastq.gz -I raw/sample_R2.fastq.gz \
  -o trim/sample_R1.fq.gz  -O trim/sample_R2.fq.gz \
  --detect_adapter_for_pe \
  --trim_poly_g --poly_g_min_len 10 \
  -q 15 -u 40 -n 5 -l 30 \
  --overrepresentation_analysis \
  --json qc/sample.fastp.json \
  --html qc/sample.fastp.html \
  --report_title "sample WGS 30x" \
  -w 8 2> logs/sample.fastp.log

What each flag does. --detect_adapter_for_pe turns on adapter auto-detection for paired-end input, which fastp does not do by default (single-end detection is on by default). For paired reads it also infers adapters from read overlap, which is more reliable than sequence matching against a known list. -q 15 sets the quality threshold for a base to count as qualified and -u 40 discards a read if more than 40 percent of its bases fall below that. -n 5 drops reads with more than five ambiguous N bases. -l 30 drops reads shorter than 30 bp after trimming, which keeps unalignable fragments out of the BAM.

Two flags we deliberately omit. Do not use --dedup on WGS. Duplicate identification belongs after alignment, where samtools markdup or Picard can use mapping coordinates and mate positions instead of raw sequence identity. Sequence-level deduplication at 30x coverage will remove genuinely independent fragments that happen to start at the same position. Do not use --cut_right unless the report shows a clear quality collapse at the 3’ end. Sliding-window trimming is the right tool for a bad run. It is not routine hygiene.

If your reads come from a two-color instrument and the raw report shows a G spike at the 3’ end, confirm after trimming that it is gone. If you see instead a long stretch of very low quality across all cycles in one tile range, the run has a problem that trimming cannot fix, and you should ask the lab for the InterOp metrics.

4. Adjust for RNA-seq

RNA-seq libraries have shorter inserts and different artifacts, so the flag set changes. Poly-A carryover from oligo-dT priming shows up as poly-A tails on read ends, and adapter read-through is common when the insert is shorter than the read length.

fastp \
  -i raw/rna_R1.fastq.gz -I raw/rna_R2.fastq.gz \
  -o trim/rna_R1.fq.gz  -O trim/rna_R2.fq.gz \
  --detect_adapter_for_pe \
  --trim_poly_x --poly_x_min_len 10 \
  --trim_poly_g \
  -q 15 -u 40 -l 25 \
  --overrepresentation_analysis \
  --json qc/rna.fastp.json --html qc/rna.fastp.html \
  -w 8

--trim_poly_x removes homopolymer tails of any base, which covers poly-A and poly-T. Keep the minimum length at 10 so you do not trim genuine sequence. Lower -l to 25 because splice-aware aligners and pseudoaligners such as Salmon and kallisto still place short reads usefully, and because a read that is too short simply fails to pseudoalign rather than producing a wrong answer.

Do not enable base correction (--correction) for RNA-seq without thinking. Base correction uses overlapping paired reads to fix mismatches, which works well for DNA. For RNA it can erase real allele-specific expression signal at heterozygous sites inside the overlap region. If your goal is expression quantification followed by differential analysis, the standard practice of adapter and quality preprocessing before quantification applies2, and gene-level counts are largely insensitive to the fine trimming parameters.

5. Handle UMIs before anything else touches the reads

If your library used unique molecular identifiers, they usually sit at the start of read 1 or in a separate index read. fastp can move them into the read name so that a downstream deduplicator can find them.

fastp \
  -i raw/s_R1.fastq.gz -I raw/s_R2.fastq.gz \
  -o trim/s_R1.fq.gz   -O trim/s_R2.fq.gz \
  --umi --umi_loc read1 --umi_len 8 --umi_skip 2 \
  --detect_adapter_for_pe \
  --json qc/s.fastp.json --html qc/s.fastp.html -w 8

--umi_loc read1 takes the UMI from the 5’ end of read 1, --umi_len 8 sets its length, and --umi_skip 2 discards a two-base spacer after it. The UMI is appended to the read name after a colon. Verify with zcat trim/s_R1.fq.gz | head -1 that the tag appears where your deduplication tool expects it. Getting the length wrong by one base is a common and silent error: the reads still process, and the duplicate rate afterwards looks implausible.

6. Read the JSON and do not rely on the HTML

The HTML report is good for one sample and useless for twenty. The JSON has the same content in a form you can query, and this is where the value of fastp for a longitudinal profile lies. Pull a fixed set of fields per sample and append them to a table you keep forever.

for f in qc/*.fastp.json; do
  s=$(basename "$f" .fastp.json)
  jq -r --arg s "$s" '[
    $s,
    .summary.before_filtering.total_reads,
    .summary.after_filtering.total_reads,
    .summary.before_filtering.q30_rate,
    .summary.after_filtering.q30_rate,
    .summary.after_filtering.gc_content,
    .duplication.rate,
    .insert_size.peak,
    .adapter_cutting.adapter_trimmed_reads
  ] | @tsv' "$f"
done > qc/summary.tsv

Read the columns as follows. A drop of more than a few percent between before_filtering.total_reads and after_filtering.total_reads means the quality filter is doing real work, and you should look at why. q30_rate below about 0.85 on a patterned flow cell suggests a run problem rather than a sample problem. insert_size.peak below your read length tells you adapter read-through is expected, and adapter_trimmed_reads should then be a large fraction of the total. GC content far from the expected value for the library type points at contamination or an unexpected input.

The filtering_result block breaks down exactly why reads were dropped, with counts for low_quality_reads, too_many_N_reads, too_short_reads, and low_complexity_reads. Check the balance. If too_short_reads dominates, your insert size distribution is the story. Read quality is not the issue.

7. Run the whole batch and keep provenance

fastp scales to about 16 threads before the gzip output path saturates, so -w 16 is the practical ceiling. Running several samples in parallel beats giving one sample more threads, as long as your disk keeps up. Handling many samples at once is where preprocessing cost becomes a real engineering concern rather than an afterthought3.

ls raw/*_R1.fastq.gz | sed 's|raw/||; s|_R1.fastq.gz||' > samples.txt

parallel -j 4 --joblog logs/fastp.joblog '
  fastp -i raw/{}_R1.fastq.gz -I raw/{}_R2.fastq.gz \
        -o trim/{}_R1.fq.gz -O trim/{}_R2.fq.gz \
        --detect_adapter_for_pe --trim_poly_g \
        -q 15 -u 40 -l 30 --overrepresentation_analysis \
        --json qc/{}.fastp.json --html qc/{}.fastp.html \
        -w 4 2> logs/{}.fastp.log
' :::: samples.txt

Note the per-sample --json and --html paths. Without them fastp writes fastp.json and fastp.html into the working directory, and four parallel jobs will overwrite each other’s reports. This is the single most common way people lose their QC metrics.

Afterwards, record the environment alongside the outputs.

{ fastp --version; date -Iseconds; md5sum raw/*.fastq.gz; } > qc/provenance.txt

Multi-omics studies that combine sequencing with other measurement layers depend on this kind of consistent upstream preprocessing before the layers can be compared4. Your own profile is the same problem at n equals one.

8. Verify you did not break anything

Three checks, in order. First, confirm read counts match between the trimmed R1 and R2 files, because a mismatch will make aligners fail or silently misbehave.

for r in R1 R2; do
  echo -n "$r "
  zcat trim/sample_${r}.fq.gz | awk 'END{print NR/4}'
done

Second, run fastp on the trimmed output in report-only mode and confirm that the adapter content is near zero and the poly-G spike is gone. Third, align a small subset and compare the mapping rate against an untrimmed subset.

seqtk sample -s42 trim/sample_R1.fq.gz 200000 > sub_R1.fq
seqtk sample -s42 trim/sample_R2.fq.gz 200000 > sub_R2.fq
bwa-mem2 mem -t 8 ref.fa sub_R1.fq sub_R2.fq | samtools flagstat -

If trimming lowered your mapping rate, you trimmed too hard. Back off the quality filter first.

Common problems

Reports overwriting each other. Always pass --json and --html with a per-sample path, as described above.

Output not compressed. fastp decides on gzip from the output filename extension. If you write -o sample_R1.fq you get a plain-text file several times larger. Use .gz, and set -z between 1 and 9 to trade compression level against speed. Level 4 is a reasonable default for intermediates.

Error: sequence and quality have different length. This almost always means a truncated gzip file from an interrupted download. Test with gzip -t raw/*.fastq.gz before you blame the tool, and re-fetch anything that fails.

Paired files out of order. fastp assumes R1 and R2 records appear in the same order. Files that were filtered separately by some earlier step will produce nonsense. Re-pair them with seqkit pair or fastq_pair before running fastp.

Memory growth with --dedup. Deduplication builds a hash table over reads, and --dup_calc_accuracy from 1 to 6 controls its size. Level 6 on a 30x genome will use tens of gigabytes. If you only want the duplication estimate and not the filtering, leave --dedup off, since fastp reports an estimated duplication rate anyway. Use --dont_eval_duplication to skip even that and save time.

Threads above 16 not helping. The parallel section of fastp is the per-read processing, and beyond roughly 16 workers the gzip reader and writer dominate. Run more samples concurrently instead.

Overrepresented sequences that look alarming. --overrepresentation_analysis lists frequent subsequences by position. Highly expressed transcripts, rRNA, and mitochondrial sequence will show up in RNA-seq and are expected. Blast an unexplained sequence before assuming contamination. These per-sample statistics are inputs to later analysis, and downstream methods behave better when the noisy and artifactual features are removed early5.

Finally, a boundary worth stating plainly. Everything here is measurement and file handling. Nothing in a fastp report supports any conclusion about your health. Any interpretation of variants, expression, or biomarkers that bears on a medical question belongs with a licensed clinician or a certified clinical laboratory.

Footnotes

  1. Maria V. Schneider, Sandra Orchard. Omics Technologies, Data and Bioinformatics Principles. Methods in Molecular Biology, 2011. https://doi.org/10.1007/978-1-61779-027-0_1 ↩

  2. Moritz Kebschull, Melanie Julia Fittler, Ryan T. Demmer, et al. Differential Expression and Functional Analysis of High-Throughput -Omics Data Using Open Source Tools. Methods in Molecular Biology, 2016. https://doi.org/10.1007/978-1-4939-6685-1_19 ↩

  3. Bonnie Berger, Jian Peng, Mona Singh. Computational solutions for omics data. Nature Reviews Genetics, 2013. https://doi.org/10.1038/nrg3433 ↩

  4. Kunhao Xie, Yixuan Chen, Xiaoqin Wang, et al. Physiological and multi-omics analysis revealed the mechanism of arbuscular mycorrhizal fungi to cadmium toxicity in green onion. Ecotoxicology and Environmental Safety, 2025. https://doi.org/10.1016/j.ecoenv.2025.117754 ↩

  5. Yasset Perez-Riverol, Max Kuhn, Juan Antonio Vizcaíno, et al. Accurate and fast feature selection workflow for high-dimensional omics data. PLOS ONE, 2017. https://doi.org/10.1371/journal.pone.0189875 ↩

Pothos builds longitudinal molecular profiles of individuals: whole-genome sequencing, single-cell RNA sequencing, and plasma proteomics, delivered as a detailed report, every data file, and stored cells. Build your profile.