TL;DR

• FASTA records use > headers followed by sequence lines.
• Most tools use the first header token as the sequence ID.
• Keep IDs unique and sequence lines clean (no spaces/symbol noise).
• samtools faidx enables fast indexed region extraction.
• seqkit, seqtk, and bioawk cover common filtering, sampling, and inspection tasks.

Structure

A FASTA record begins with a header line starting with > followed by one or more sequence lines.

>seq1 Homo sapiens example
ATGCGTACGTAGCTAGCTAG

Most FASTA files contain many records:

>chr1
NNNNATGCGT...
>chr2
TTGCAAGT...

Practical Conventions

• Header lines (>...) are record identifiers plus optional description.
• Sequence is typically uppercase letters (A/C/G/T/N for DNA; amino-acid alphabet for proteins).
• Line wrapping is common (for example 60 or 80 chars/line), but not required by many tools.
• Large references are often distributed as compressed files (.fa.gz / .fasta.gz).

Common Pitfalls

• Duplicate sequence IDs in headers can break downstream tools.
• Unexpected characters in sequence lines (spaces, digits, symbols) cause parser errors.
• Some tools only use the first token in the header as the ID (before first whitespace).

Common Uses

• Reference genomes
• Transcript/protein databases
• Input for alignment and search tools

Useful FASTA Tools

seqkit

Fast toolkit for everyday FASTA/FASTQ operations.

# stats summary
seqkit stats sequences.fasta

# filter by minimum sequence length
seqkit seq -m 1000 sequences.fasta > sequences.min1k.fasta

samtools faidx

Index FASTA and pull subregions quickly.

# build FASTA index (.fai)
samtools faidx reference.fasta

# extract region
samtools faidx reference.fasta chr1:1000-2000

seqtk

Lightweight FASTA/FASTQ processing.

# sample 100 sequences (reproducible with seed)
seqtk sample -s42 sequences.fasta 100 > subset.fasta

bioawk

AWK-style filtering with FASTA awareness.

# print IDs and sequence lengths
bioawk -c fastx '{print $name, length($seq)}' sequences.fasta

TL;DR

• FASTQ uses 4 lines per read: header, sequence, separator, quality.
• Sequence and quality strings must be the same length.
• Most modern pipelines assume Sanger/Illumina 1.8+ encoding (Phred+33).
• FASTQ is usually compressed as .fastq.gz.
• fastqc, fastp, seqkit, and seqtk are common day-to-day tools.

Structure

Each read is represented by 4 lines:

1. @ header
2. sequence
3. + separator
4. quality string

@read1
ACGTACGTACGT
+
IIIIIIIIIIII

Most FASTQ files contain millions of repeated 4-line records.

Practical Conventions

• Header lines begin with @ and may include instrument/run metadata.
• + line may repeat the read ID or be just +.
• Quality characters encode Phred scores (commonly Phred+33).
• Files are often gzip-compressed (.fastq.gz) to reduce storage.

Common Pitfalls

• Sequence and quality lengths not matching (invalid record).
• Mixing quality encodings (older Phred+64 vs modern Phred+33).
• Truncated files from interrupted transfers/downloads.
• Paired-end files getting out of sync (R1 and R2 order mismatch).

Common Uses

• Raw sequencing output
• Read-level QC
• Input for alignment and assembly pipelines

Useful FASTQ Tools

fastqc

Standard read-level quality control report.

fastqc sample_R1.fastq.gz sample_R2.fastq.gz

fastp

Fast all-in-one filtering/trimming with QC outputs.

fastp \
  -i sample_R1.fastq.gz -I sample_R2.fastq.gz \
  -o sample_R1.clean.fastq.gz -O sample_R2.clean.fastq.gz \
  -h fastp.html -j fastp.json

seqkit

Convenient FASTA/FASTQ stats and filtering.

# summary stats
seqkit stats sample.fastq.gz

# keep reads with minimum length 75
seqkit seq -m 75 sample.fastq.gz > sample.min75.fastq

seqtk

Lightweight toolkit for sampling and format conversion.

# subsample reads reproducibly
seqtk sample -s42 sample.fastq.gz 100000 > subset.fastq

TL;DR

• BED uses 0-based, half-open coordinates: [start, end).
• Minimum valid BED record is 3 fields: chrom, start, end.
• BED6 (+ name, score, strand) is a common interoperable subset.
• BED12 extends BED for exon/block models (for example transcripts).
• bedtools is the standard toolkit for overlap, merge, subtract, and coverage operations.
• gxf2bed is useful for converting GTF/GFF annotations into BED coordinates.

Structure

A BED file is plain text, one interval per line, tab-delimited.

Required BED3 fields:

1. chromosome
2. start (0-based)
3. end (half-open)

Simple BED3 example:

chr1	1000	2000
chr1	2500	2600
chr2	150	400

Common BED6 example:

chr1	1000	2000	peak_001	850	+
chr1	2500	2600	peak_002	420	-

BED12 adds block structure (used for transcript/exon-style annotations), including fields like thickStart, thickEnd, itemRgb, blockCount, blockSizes, and blockStarts.

Coordinate model (important)

BED uses 0-based start and end-exclusive coordinates.

• Interval chr1 100 200 includes bases 101..200 in 1-based closed notation.
• Length is end - start (here: 100 bp).

Practical Conventions

• Use tabs, not spaces, between columns.
• Keep chromosome naming consistent (chr1 vs 1) across all files in a workflow.
• Ensure start < end for every interval.
• Use sorted BED for reproducible downstream behavior.
• Prefer BED6 when strand-aware operations are needed.

Common Pitfalls

• Mixing coordinate systems (BED 0-based vs GFF/GTF often 1-based).
• Unsorted BED files causing unexpected merge/intersect results.
• Inconsistent chromosome naming between BED and reference/alignment files.
• Invalid intervals (start >= end) or negative starts.
• Assuming all tools interpret optional columns the same way.

Common Uses

• Region annotation
• Peak calls
• Intersections and interval operations

Useful BED Tools

bedtools sort

Sort BED intervals for stable downstream processing.

bedtools sort -i regions.bed > regions.sorted.bed

bedtools intersect

Find overlaps between two interval sets.

bedtools intersect \
  -a peaks.sorted.bed \
  -b genes.sorted.bed \
  -wa -wb > peaks_in_genes.tsv

bedtools merge

Merge overlapping or adjacent intervals.

bedtools merge -i regions.sorted.bed > regions.merged.bed

bedtools coverage

Compute coverage of features over targets.

bedtools coverage \
  -a targets.sorted.bed \
  -b alignments.bed > target_coverage.tsv

gxf2bed

Convert GTF/GFF-style annotation files to BED.

# convert GTF to BED
gxf2bed annotations.gtf > annotations.bed

# convert GFF3 to BED
gxf2bed annotations.gff3 > annotations.bed

TL;DR

• BAM is binary/compressed SAM, so it is smaller and faster to process at scale.
• Coordinate-sorted BAM plus index (.bai or .csi) enables fast random-access region queries.
• Core SAM fields (especially CIGAR) describe how each read aligns to the reference.
• Optional tags carry rich metadata, including alignment metrics (NM, MD, AS) and modified bases (MM, ML).
• RG tags on reads map to header read groups, where SM defines the sample name used by many downstream tools.
• samtools is the standard toolkit for sorting, indexing, filtering, and summary stats.

Structure

A BAM file contains:

1. Header (SAM-style metadata, binary encoded)
2. Alignment records (one per read/alignment)

Conceptually, BAM represents the same fields as SAM (QNAME, FLAG, RNAME, POS, MAPQ, CIGAR, etc.), but in compressed binary form.

Minimal SAM-style example:

@HD	VN:1.6	SO:coordinate
@SQ	SN:chr1	LN:248956422
read0001	99	chr1	10001	60	10M1I15M2D24M	=	10120	180	ACGTT...	IIII...	NM:i:3

• @HD and @SQ are header lines (format version, sort order, reference dictionary).
• The alignment line is tab-delimited and includes core fields plus optional tags (for example NM:i:3).

CIGAR quick explanation

The CIGAR string encodes how the read aligns to the reference.

• M: alignment match/mismatch block
• I: insertion in read relative to reference
• D: deletion from read relative to reference
• S: soft clipping (bases present in read, not aligned)
• H: hard clipping (bases removed from stored sequence)
• N: skipped region on reference (common in RNA-seq introns)

For 10M1I15M2D24M:

• 10 aligned bases, then 1 inserted base, then 15 aligned bases, then a 2-base deletion on the reference, then 24 aligned bases.

Modified bases in SAM/BAM tags

Modified bases are typically stored in optional tags, most commonly:

• MM: modification type + positions (delta encoded)
• ML: per-site modification probabilities (byte scale)

Example (schematic):

read0002	0	chr1	20501	50	30M	*	0	0	ACGTC...	IIII...	MM:Z:C+m,5,12;	ML:B:C,220,180

• C+m means 5mC calls on cytosines.
• Position list (5,12) indicates candidate modified-base offsets along the read for that base type.
• ML values (here 220,180) are confidence/probability values corresponding to the MM calls.

Exact interpretation can vary by basecaller/tool version, so always confirm against your caller’s spec when parsing modified-base tags.

RG and SM tags (sample metadata)

• RG:Z:<id> appears on alignment records and points to a read group ID.
• SM:<sample_name> appears in header @RG lines and defines the biological sample for that read group.

Minimal header + record example:

@RG	ID:rg001	SM:NA12878	PL:ILLUMINA
read0003	99	chr1	30001	60	50M	=	30100	149	ACGT...	IIII...	RG:Z:rg001

In practice, many downstream tools use SM for per-sample aggregation and RG for lane/library/platform-aware processing (for example duplicate marking and BQSR).

Practical Conventions

• BAM files are usually coordinate-sorted for downstream analysis workflows.
• Indexed BAM (samtools index) is expected by genome browsers and region-based tools.
• Headers often include @SQ (reference names/lengths), @RG (read groups), and @PG (pipeline provenance).
• Large or highly fragmented references may use .csi indexes instead of .bai.

Header sanity checks

Before downstream analysis, header inspection is a fast sanity check for common integration issues.

• @HD confirms sort order (SO:coordinate expected in many workflows).
• @SQ verifies contig names and lengths against your reference FASTA.
• @RG confirms read-group/sample metadata (for example SM) is present.
• @PG provides pipeline provenance (what tools/versions touched the BAM).

# inspect BAM header sections
samtools view -H sample.bam | rg '^@HD|^@SQ|^@RG|^@PG'

# quick check for coordinate-sorted header
samtools view -H sample.bam | rg '^@HD'

Why indexed BAM is useful

• Fast random access to specific loci/regions without scanning the entire file.
• Lower I/O and compute for region-based QC, visualization, and variant workflows.
• Enables interactive browsing in tools like IGV/JBrowse with near-instant jumps.
• Essential for cloud/remote workflows where minimizing transferred bytes matters.

Common Pitfalls

• Using an unsorted BAM where a sorted BAM is required (causes tool errors or wrong assumptions).
• Missing or stale index after replacing/updating a BAM file.
• Header/reference mismatch between BAM and reference FASTA used downstream.
• Unexpected duplicate/secondary/supplementary alignments when counting reads naively.
• Ignoring mapping quality (MAPQ) or duplicate flags in variant/QC analyses.

Common Uses

• Read alignment storage
• Variant calling workflows
• Coverage and mapping quality analysis

Useful BAM Tools

samtools view

Inspect, filter, and convert between BAM/SAM/CRAM.

# quick header + first alignments (SAM text view)
samtools view -h sample.bam | head

# keep mapped primary alignments with MAPQ >= 20
samtools view -b -F 0x904 -q 20 sample.bam > sample.mapq20.primary.bam

samtools sort

Sort BAM records (typically by coordinate).

samtools sort -o sample.sorted.bam sample.bam

samtools index

Build BAM index for random access.

samtools index sample.sorted.bam

samtools flagstat

Generate quick mapping/alignment summary statistics.

samtools flagstat sample.sorted.bam

cramino

Fast long-read alignment QC summaries (commonly used with ONT BAM/CRAM files).

# generate a basic cramino report from a BAM
cramino sample.sorted.bam > sample.cramino.tsv

nanocov

Per-base coverage statistics and plots from BAM files.

# basic run
nanocov \
  --input sample.sorted.bam \
  --output-dir nanocov_out

# coverage on target regions from BED
nanocov \
  --input sample.sorted.bam \
  --bed targets.bed \
  --output-dir nanocov_targets \
  --prefix sample1

TL;DR

• VCF is a tab-delimited text format for sequence variants.
• Core columns are fixed (CHROM, POS, ID, REF, ALT, QUAL, FILTER, INFO), with optional sample genotype columns.
• Coordinates are 1-based, and complex alleles are anchored with a left reference base.
• Most production workflows use bgzipped VCF (.vcf.gz) with a tabix index (.tbi/.csi).
• bcftools is the standard toolkit for filtering, querying, normalization, and stats.

Structure

A VCF file includes:

• metadata lines beginning with ##
• one header line beginning with #CHROM
• one row per variant record

Minimal example:

##fileformat=VCFv4.3
##contig=<ID=chr1,length=248956422>
##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	sample1
chr1	10177	rs367896724	A	AC	100	PASS	DP=14	GT	0/1

Fixed columns

1. CHROM: chromosome/contig
2. POS: 1-based position
3. ID: variant identifier (or .)
4. REF: reference allele
5. ALT: alternate allele(s), comma-separated if multiallelic
6. QUAL: variant quality (or .)
7. FILTER: PASS or filter labels
8. INFO: semicolon-delimited key-value annotations

Optional sample columns begin with FORMAT, then one column per sample.

Genotype fields

FORMAT defines per-sample subfields (for example GT:DP:AD:GQ).

Example:

#CHROM	POS	ID	REF	ALT	QUAL	FILTER	INFO	FORMAT	s1	s2
chr2	200123	.	G	A	60	PASS	DP=52	GT:DP:AD:GQ	0/1:25:12,13:99	0/0:27:27,0:99

• GT: genotype (0/0, 0/1, 1/1, ./.)
• DP: sample read depth
• AD: allele depths (REF,ALT1,ALT2,...)
• GQ: genotype quality

Coordinate and allele conventions

• SNP example: REF=A, ALT=G at POS.
• Insertion/deletion alleles are left-anchored by convention.
• For indels and multiallelic sites, normalization (left-align + split) is often required before comparison/merge.

Practical Conventions

• Compress and index VCFs for fast random access: .vcf.gz + .tbi/.csi.
• Keep contig names consistent across reference, BAM/CRAM, and VCF files.
• Use normalized, decomposed records for robust comparisons.
• Preserve full metadata (##INFO, ##FORMAT, ##FILTER) when transforming files.
• Validate sample order and identity before cohort merges.

Header sanity checks

VCF headers are a quick preflight check before filtering, merging, or annotation.

• ##fileformat confirms parser compatibility expectations.
• ##contig should match your reference naming/length conventions.
• ##INFO, ##FORMAT, and ##FILTER definitions should exist for used fields.
• #CHROM ... sample columns should match expected sample IDs/order.

# inspect key header metadata
bcftools view -h input.vcf.gz | rg '^##(fileformat|contig|INFO|FORMAT|FILTER)'

# list sample names in header order
bcftools query -l input.vcf.gz

Common Pitfalls

• Mixing chr and non-chr contig naming across inputs.
• Comparing unnormalized indels and getting false discordance.
• Dropping header metadata during manual edits/reformatting.
• Misinterpreting missing values (.) vs true zero values.
• Treating QUAL as equivalent to genotype quality (GQ).

Common Uses

• SNP/indel call representation
• Variant filtering and annotation
• Cohort and population analyses

Useful VCF Tools

Header provenance (run logs)

From the 2 tools listed below (bcftools and tabix), 1 tool commonly writes command provenance into VCF headers when producing VCF/BCF output:

• bcftools (for example view, norm)

They typically add lines such as ##bcftoolsVersion and ##bcftoolsCommand. Within bcftools, submodules such as query and stats usually produce tabular/text outputs (not VCF headers), and tabix creates indexes without editing header metadata.

# inspect provenance lines in a VCF header
bcftools view -h input.vcf.gz | rg '^##(bcftools|source)'

bcftools (view, query, norm, stats)

bcftools is one tool with multiple subcommands for filtering, querying, normalization, and QC.

bcftools view

Filter and subset variant records.

# keep PASS variants only
bcftools view -f PASS input.vcf.gz -Oz -o pass.vcf.gz

# keep only biallelic SNPs
bcftools view -m2 -M2 -v snps input.vcf.gz -Oz -o snps.biallelic.vcf.gz

bcftools query

Extract tabular fields for reporting.

bcftools query -f '%CHROM\t%POS\t%REF\t%ALT\t%QUAL\n' input.vcf.gz > variants.tsv

bcftools norm

Normalize and split multiallelic records.

bcftools norm \
  -f reference.fasta \
  -m -any input.vcf.gz -Oz -o input.norm.vcf.gz

bcftools stats

Generate summary statistics for QC.

bcftools stats input.vcf.gz > input.vcf.stats.txt

tabix

Index bgzipped VCF for region queries.

tabix -p vcf input.vcf.gz

TL;DR

• Both formats use 1-based, closed genomic coordinates.
• Core records are 9 columns: seqid, source, type, start, end, score, strand, phase, attributes.
• GFF3 uses key=value attributes and explicit parent/child IDs (ID, Parent).
• GTF commonly uses key "value"; attributes, especially gene_id and transcript_id.
• Prefer GFF3 for hierarchical feature models and standards compliance; GTF is common in RNA-seq pipelines.

Structure

Each non-comment line has 9 tab-delimited fields:

1. seqid (chromosome/contig)
2. source
3. feature type (for example gene, transcript, exon, CDS)
4. start (1-based)
5. end (1-based, inclusive)
6. score (. if missing)
7. strand (+, -, or .)
8. phase (0, 1, 2 for CDS, else .)
9. attributes

GFF3 example:

##gff-version 3
chr1	RefSeq	gene	11869	14409	.	+	.	ID=gene1;Name=DDX11L1
chr1	RefSeq	mRNA	11869	14409	.	+	.	ID=tx1;Parent=gene1
chr1	RefSeq	exon	11869	12227	.	+	.	ID=exon1;Parent=tx1

GTF example:

chr1	ENSEMBL	gene	11869	14409	.	+	.	gene_id "GENE1"; gene_name "DDX11L1";
chr1	ENSEMBL	transcript	11869	14409	.	+	.	gene_id "GENE1"; transcript_id "TX1";
chr1	ENSEMBL	exon	11869	12227	.	+	.	gene_id "GENE1"; transcript_id "TX1"; exon_number "1";

Coordinate model (important)

GFF3/GTF are 1-based and end-inclusive.

• Interval chr1 11869 12227 has length 12227 - 11869 + 1 = 359 bp.
• Converting to BED requires coordinate shift: BED_start = start - 1, BED_end = end.

Practical Conventions

• Keep feature hierarchy consistent (gene -> transcript -> exon/CDS).
• Use stable feature IDs (especially in GFF3 ID and Parent).
• Keep chromosome naming consistent across FASTA/BAM/VCF/BED.
• Use tabs only; spaces in attributes should remain inside quoted values.
• Sort by chromosome and start when possible for reproducible processing.

Common Pitfalls

• Mixing 1-based GFF3/GTF coordinates with 0-based BED coordinates.
• Broken parent-child relationships (missing Parent/ID, inconsistent transcript IDs).
• Invalid or inconsistent attributes that downstream parsers cannot interpret.
• Treating GFF3 and GTF attribute syntax as interchangeable.
• Incorrect CDS phase values causing translation/frame issues.

Common Uses

• Gene and transcript annotation
• Feature counting and RNA-seq quantification input
• Region extraction and annotation conversion (for example to BED)

Useful Annotation Tools

gffread

Validate, filter, and convert GFF/GTF annotations.

# basic validation-style parse and export
gffread annotations.gff3 -T -o annotations.gtf

# convert GTF to GFF3
gffread annotations.gtf -o annotations.gff3

gffcompare

Compare transcript annotations against a reference.

gffcompare -r reference.gtf -o compare_out query.gtf

gffutils

Build/query a feature database from GFF/GTF.

# create sqlite DB from GFF3
python -m gffutils.cli create annotations.gff3 --db annotations.db

gxf2bed

Convert GFF/GTF annotations to BED intervals.

gxf2bed annotations.gff3 > annotations.bed

TL;DR

• POD5 stores raw electrical signal traces from ONT sequencing runs.
• It replaces FAST5 in many modern ONT workflows with better performance and simpler access patterns.
• A POD5 file typically includes read IDs, signal chunks, timing/scaling info, and run context metadata.
• Basecalling tools (for example Dorado) use POD5 as direct input.
• pod5 CLI tools are used to inspect, subset, and convert POD5 datasets.

Structure

POD5 is a binary container format (not line-based text like FASTQ/VCF/BED).

Conceptually, it stores:

1. Read records (read_id and per-read metadata)
2. Raw signal arrays (current levels across time)
3. Calibration/scaling fields (for signal interpretation)
4. Run/context metadata (flowcell, run identifiers, acquisition details)

Unlike FASTQ (basecalled sequence) or BAM (aligned reads), POD5 captures pre-basecalling raw signal data.

Practical Conventions

• Keep POD5 files immutable once generated to preserve provenance.
• Track software/basecaller version alongside POD5 datasets.
• Organize files by run and sample metadata for downstream traceability.
• Use checksums when moving POD5 across storage systems.
• Convert/subset with official tooling rather than ad-hoc binary manipulation.

Common Pitfalls

• Treating POD5 as if it were sequence-level output (it is signal-level data).
• Losing run metadata linkage when splitting files without consistent naming.
• Mixing POD5 batches from different chemistry/basecaller expectations without tracking metadata.
• Underestimating storage and I/O requirements for raw signal datasets.
• Attempting manual parsing without format-aware libraries/tools.

Common Uses

• Input to ONT basecalling workflows
• Modified-base and signal-level analyses
• Archival of raw nanopore run data

Useful POD5 Tools

pod5 inspect

Inspect summary metadata for POD5 files.

pod5 inspect reads.pod5

pod5 view

View selected records/fields from POD5 datasets.

pod5 view reads.pod5 --ids read_ids.txt

pod5 subset

Create a smaller POD5 from selected reads.

pod5 subset reads.pod5 --ids read_ids.txt --output subset.pod5

dorado

Use POD5 directly as basecalling input.

dorado basecaller hac reads.pod5 > basecalls.bam

Note: SLOW5/BLOW5 are third-party community formats and are not officially supported by Oxford Nanopore Technologies (ONT).

TL;DR

• SLOW5 is a human-readable text format for nanopore raw signal data.
• BLOW5 is the binary compressed form of SLOW5, optimized for performance and storage.
• Both formats are designed to improve I/O efficiency compared with legacy FAST5 workflows.
• Typical records include read identifiers, run metadata, and raw signal vectors.
• slow5tools is the core toolkit for inspect, conversion, merge/split, and indexing operations.

Structure

SLOW5 records are tab-delimited text with:

1. Header/meta lines (global run metadata and field definitions)
2. One line per read record
3. Core per-read fields (for example read ID, digitisation/range/offset, sampling rate)
4. Raw signal values

BLOW5 stores the same logical content in binary compressed form.

In practice:

• Use SLOW5 when readability/interchange in text form is useful.
• Use BLOW5 for large-scale processing, lower disk footprint, and faster random access.

Practical Conventions

• Prefer BLOW5 for production workflows and large datasets.
• Keep tool/version metadata with files for reproducibility.
• Preserve read-group/run identifiers when converting between formats.
• Index BLOW5 files when supported/needed by downstream tooling.
• Use official converters (slow5tools) instead of custom parsers for transformations.

Common Pitfalls

• Confusing text SLOW5 and binary BLOW5 in downstream command usage.
• Losing metadata consistency when merging runs from different experiments.
• Ignoring version compatibility between tools and file schema.
• Treating signal-level files as sequence-level outputs.
• Skipping integrity checks after large file transfers.

Common Uses

• Raw nanopore signal storage
• High-throughput signal processing workflows
• Conversion pipelines between nanopore raw-data formats

Useful SLOW5/BLOW5 Tools

slow5tools view

Inspect SLOW5/BLOW5 content.

slow5tools view reads.blow5 | head

slow5tools stats

Generate quick file/read summary statistics.

slow5tools stats reads.blow5

slow5tools merge

Merge multiple SLOW5/BLOW5 files.

slow5tools merge run1.blow5 run2.blow5 -o merged.blow5

slow5tools split

Split a large file into smaller chunks.

slow5tools split reads.blow5 -d split_out/

slow5tools index

Create an index for random access.

slow5tools index reads.blow5