Post

rust-lapper v2.0.0-beta.1 is published! This is a large update to mitigate the worst case scenario where there are large intervals that envelope many smaller intervals. It's a lot faster (more details below), and keeps the same api, as well as the BITS implementation. crates.io/crates/rust-...

The main change is adding 32-interval block index that allows for skipping, as well as doing simd comparisons when there are some matches present, while maintaining forward ordered iteration on query. See the PR for details: github.com/sstadick/rus...

Why beta? This is a drastic change from the original rust-lapper, and while I've tested it every way I can think of, I'd still rather not pull the rug out on people who use this and don't have appropriate version guards set up.

While the ideas were mine (and lean heavily on ideas from super-intervals), the implementation in this case was nearly 100% AI written. While I've reviewed it all, tested it all, etc, I still have mixed feelings about shipping AI generated code in a repo that was previously human-only.

Perf: This should generally be 1.5x faster than the original rust-lapper, and 76-90x faster on the worst case scenarios. Huge thank you to the bio-polars team that put out this blog post which inspired the second look at rust-lapper biodatageeks.org/polars-bio/b...

I’ve generally defaulted to coitrees crate, any updated comparisons to that approach? (There was a dead link off the GitHub repo that may have hosted it)

Big caveat - rust-lapper won't work nicely for insertions. But on the datasets in that polars article, lapper-v2 is 1.35x to 1.8x faster than coitrees, including construction time. Thanks for pointing out that link, fixing it!

Because I'm feeling particularly AI conscious right now, just want to say that "Big caveat" was actually my own wording here. I'm that original it seems 🫥