Nils Homer

Nils Homer

@nilshomer.com

I write software for Biologists. Founder @fulcrumgenomics, Bioinformatician, Computer Scientist, Coder, Father of 2xGirls. Opinions are my own.

2.3K followers215 following173 posts

bwa-mem3 is now 6-8x faster than the original, and now as fast as minibwa. What's stopping you from using it?

bwa-mem3 v0.12.0 is out 🧬 Since v0.10.0: faster on both Arm & x86 (--fast is now ~2Γ— minibwa and stock is within ~15% on x86) while staying a byte-identical (--compat). It also brings ~30–40% faster methylation and a big memory-safety pass. github.com/fg-labs/bwa-... #bioinformatics #genomics

github.comRelease v0.12.0 Β· fg-labs/bwa-mem30.12.0 β€” faster on both architectures, a faster --meth, and a large safety pass A broad optimization release that speeds up both Arm and x86 (unlike 0.11.0, which concentrated on Arm), plus a methy...

fgumi v0.7.0 is released: github.com/fulcrumgenom... 1. sort is 28% faster; 2-3x faster than samtools 2. improved CODEC consensus calling 3. dedup now outputs metrics closer to picard MarkDuplicates and dupblaster 4. retag is a new tool that can copy/move/delete SAM tags

github.comRelease v0.7.0 Β· fulcrumgenomics/fgumiFor those running the fgumi command line tools v0.7.0 makes sort faster, hardens CODEC consensus, and brings dedup metrics to Picard/dupblaster parity. Sort is up to 28% faster β€” and 2–3Γ— faster th...

Want to see a mistake I made?

Fulcrum Genomics@fulcrumgenomics.com

Did you know most clinical genomics runs on research-grade code? That code has carried the field far, but benchmarking once isn't the same as verifying across versions, edge cases, & clinical use. @nilshomer.com on tool-level verification becoming routine: blog.fulcrumgenomics.com/p/benchmarki...

Nils Homer reposted

2.3 billion read pairs of 10x Flex v2 (281 GB of gzipped FASTQ) mapped in under 2 minutes on one machine (-t 64) with piscem-rs. That's ~20M read pairs/sec at ~87% mapped. One interesting part is the mapper, but what I want to talk about here is who gets the threads. 1/8

> then around 850-950 ad, their ceremonial centers were suddenly abandoned, and researchers still have no idea why

When running benchmarks on shared tenancy systems (e.g. AWS), it's hard to compare cpu/wall times over time, even with many replicates.Β  have seen performance differences up to 20% just based on fighting others for access to the memory sub-system.

bwa-mem3 v0.8.0 is out. New --compat=bwa-mem2: byte-identical to bwa-mem2 v2.2.1, verified across 1.57B records on 6 CPU types β€” including ARM. Also ~13% faster, and methylation NM/MD no longer count bisulfite conversions as mismatches. github.com/fg-labs/bwa-mem3/releases/tag/v0.8.0

github.comRelease v0.8.0 Β· fg-labs/bwa-mem30.8.0 β€” a verified drop-in for bwa-mem2, ~13% faster, and corrected methylation tags 1. --compat=bwa-mem2 β€” swap in bwa-mem3 and get the same BAM The big one. Run bwa-mem3 mem --compat=bwa-mem2 ......
View full thread

Note: bwa-mem2: doesn't reproduce itself across thread counts. Insert-size statistics are estimated per batch, and batch size scales with -t, so the same input at -t 16 and -t 32 gives you slightly different BAMs. Under --compat we pin the batch, so thread count stops affecting your results.

The rest of the release is speed β€” about 13% off the default path, 18% with --fast, and nothing got slower. And for methylation: NM and MD were counting bisulfite conversions as mismatches, inflating edit distance on nearly every read. They aren't anymore.

bwa-mem3 v0.8.0 is out. New --compat=bwa-mem2: byte-identical to bwa-mem2 v2.2.1, verified across 1.57B records on 6 CPU types β€” including ARM. Also ~13% faster, and methylation NM/MD no longer count bisulfite conversions as mismatches. github.com/fg-labs/bwa-mem3/releases/tag/v0.8.0

github.comRelease v0.8.0 Β· fg-labs/bwa-mem30.8.0 β€” a verified drop-in for bwa-mem2, ~13% faster, and corrected methylation tags 1. --compat=bwa-mem2 β€” swap in bwa-mem3 and get the same BAM The big one. Run bwa-mem3 mem --compat=bwa-mem2 ......

Add --compat=bwa-mem2 and you get exactly what bwa-mem2 v2.2.1 would have produced. Same positions, same CIGARs, same mapping qualities, same tags, same header.

We measured it rather than claiming it: 1.57 billion alignment records, across 7 datasets, on 6 CPU types, 5 replicates each. Every record matched. Zero differences.

Nils Homer reposted

Bioinformatics rewrites miss the relational impact tofolks that are still actively maintaining and developing the software. I've been guilty of this myself, and I'll be sharing my story soon so others can learn from it.

View full thread

Meritocracy and competition is great, and the moat due to implementation effort is gone, but we also have to remember the relational side of bioinformatics.

Bioinformatics rewrites miss the relational impact tofolks that are still actively maintaining and developing the software. I've been guilty of this myself, and I'll be sharing my story soon so others can learn from it.

It has involved years of late-night insights and hard fought wins. Even recently with AI, I've spent so long pouring my own ideas and discernment into it. Years diligently and genuinely supporting, maintaining and developing this software. Building a livelihood on the reputation of our work.

It's not only about speed, it's about doing it right the second time (see @tfenne.bsky.social's Riker: the successor to Picard). It's about being able to finally explore all those ideas that were left on the cutting board due to time and money constraints.

Why not byte-identical to bwa-mem2? By design. 1. bwa-mem3 breaks score ties deterministically, bwa-mem2 didn't. Reproducibility is restored. That's a feature, not a bug.

2. CPU-specific optimized code (AVX2, AVX-512, Arm) but the per-CPU (AVX2/AVX-512/Arm) kernels can score the runner-up alignment slightly differently. And since MAPQ comes from the best-vs-second-best score gap, it moves on <0.1% of reads. Plus a few extra split alignments.

Nils Homer reposted

Reporting one alignment per locus can commit a CRISPR off-target analysis to one scoring model too early. @timd.one explains how Sassy v0.2.5 enumerates every reasonable alignment while keeping runtime fast: >9Γ— as many alignments in under 30 seconds. blog.fulcrumgenomics.com/p/why-crispr...

blog.fulcrumgenomics.comWhy CRISPR Off-Target Search Should Report Multiple Alignments Per LocusHow Sassy enumerates every reasonable alignment without sacrificing runtime

Nils Homer reposted

grep is a fantastic tool, but it doesn't really work on sequencing data: it breaks when the pattern is in the read name, and doesn't support paired-end FASTQs. Thanks to @nilshomer.com, we have a new interactive guide on using fqgrep to find patterns in FASTQ files: ➑️ sandbox.bio/tutorials/fq...

sandbox.bioInteractive bioinformatics tutorialsLearn bioinformatics from your browser, no setup required. Everything runs in a sandbox, so you can experiment all you want.

What started out as "make OptiType" faster turned into a multi-month saga that discovered that "yes, a Heng Li lab tool is faster and better on the data we care about".

Part of rewriting a tool is being able to make opinionated choices about it's development, providing maintenance, and that's what I am doing here. E Pluribus Unum. #reright

πŸš€ ferro-hgvs 0.7.0 is out: our biggest release yet. Major upgrades to HGVS normalization, parsing & projection: spec-compliant 3β€² shifting, mosaic/compound alleles, multi-axis projection (g/c/n/p/r), Ensembl support, plus ~1.7Γ— faster parsing. github.com/fulcrumgenom...

github.comRelease v0.7.0 Β· fulcrumgenomics/ferro-hgvsAdded (reference) validate manifest schema/version at load, fail loud on an incompatible reference (#1003) (mosaic) parse predicted-wrapper and whole-entity-LHS =/ forms (#992) (protein) parse ins...

Nils Homer reposted

Our new genome annotation method relies almost entirely on transcriptome and alignment evidence, and as a result outperforms pretty much all other de novo pipelines. Check out the just-published paper led by Aleksey Zimin: rdcu.be/frSOg

rdcu.beEfficient evidence-based genome annotation with EviAnnNature Methods - EviAnn surpasses existing genome annotation methods by leveraging gene expression and protein sequence homology evidence to achieve higher accuracy and efficiency.

#bioinformatics re-writes are more often about doing it "right" this time, rather than faster this time. #genomics #rust #rewrites

Tim Fennell@tfenne.bsky.social

In other news I released a tool called methylsieve ~3 weeks ago, thinking it would just be a better "filter unconverted reads" tool that'd slot in between alignment and sorting. Then I got frustrated that most tooling does m-bias filtering wrong ... now I have a version 0.2 coming soon... 🧬 πŸ–₯️ 🧡

Nils Homer reposted

New riker release this morning - version 0.4.0 is out! Major updates are: - New "rna" tool that ports picard CollectRnaSeqMetrics, fgbio EstimateRnaInsertSize, and much more - New global --threads option to multithread input BAM/CRAM decoding - Big performance improvements in "wgs" and "hybcap"

πŸš„ How fast? πŸ“ We use minibwa as our speed yardstick. On WGS, --fast lands within ~3% of it on Arm/NEON (a head-to-head SIMD comparison) and in the same ballpark on x86.

πŸ” It's not byte-identical to default; --fast changes the seeding/extension heuristics. Against simulated truth, placement accuracy and recall hold (WGS within 0.06pp; high-MAPQ unchanged). Different alignments, same answers. Docs + cross-arch numbers: Β github.com/fg-labs/bwa-...

github.comdocs(allowances): authorize v0.5.0 (9dd30dd0) golden bless by nh13 Β· Pull Request #26 Β· fg-labs/bwa-mem3-benchRecords the Gate #2 sign-off for promoting v0.5.0 (9dd30dd0) to the golden, per the full bless_release run (6 arches Γ— reps=3 + holodeck accuracy + minibwa; golden gate vs v0.4.0). Golden already p...

Nils Homer reposted

New on the Fulcrum blog: minibwa, a faster mapper from @lh3lh3.bsky.social and our @nilshomer.com Its speed is great, yes, but more interesting is the decision to revisit BWA-MEM as infrastructure – keep what still works, change what limits performance, then test downstream impact. shorturl.at/xxqeI

blog.fulcrumgenomics.comMinibwa: alignment is never solvedHeng Li and Nils Homer revisit BWA-MEM with a faster mapper for short reads, accurate long reads, and bisulfite sequencing data.

Nils Homer reposted

Hey @nilshomer.com, you'd asked about this in the past: we finally got around to implementing byte-level determinism in salmon (github.com/COMBINE-lab/...). It turned out to require a lot of thought and changes, but it turns out we can do it elegantly (with RAD files) and fast (`--deterministic`)!

github.comRelease v2.2.1 Β· COMBINE-lab/salmonsalmon 2.2.1 The recommended 2.2.x release: it carries the full 2.2.0 feature set plus a security fix for a lz4_flex advisory pulled in by the new RAD compression. If you are on 2.2.0, upgrade. No ...

re-writes aren't just about speed dupblaster, riker, chelae, and methylsieve: the stack is being re-written, not just faster, but with the fixes we've always wanted

Tim Fennell@tfenne.bsky.social

Realized that I released a couple of new tools last week and didn't announce anything. The first one is *dupblaster*. Samblaster inspired streaming queryname grouped duplicate marking for NGS reads: github.com/fulcrumgenom...

minibwa + ALTs: github.com/nh13/minibwa... In-mapper ALT-aware MAPQ recovery via DRAGEN-style liftover groups. No bwa-postalt.js. Matches bwa+postalt, byte-identical without a .alt file. Works how I want it to. The real test will be when others give it a try. No warranty, implied or otherwise.

github.comALT-aware mapping via post-extension liftover groups by nh13 Β· Pull Request #7 Β· nh13/minibwaALT-aware mapping via post-extension liftover groups Draft for design feedback, opened against this fork&#39;s master. It adds optional ALT-contig awareness: given a .alt file, reads whose confiden...
This post is unavailable.