Post

Benjamin Warner
@benjaminwarner.dev

This week we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.

15 reposts74 likes

Benjamin Warner
@benjaminwarner.dev

ModernBERT-base is the first encoder to beat DeBERTaV3-base on GLUE. ModernBERT is also competitive or top scoring on single vector retrieval, ColBERT retrieval, and programming benchmarks.

Benjamin Warner
@benjaminwarner.dev

ModernBERT was designed from the ground up for speed and memory efficiency. ModernBERT is both faster and more memory efficient than every major encoder released since the original BERT.

Benjamin Warner
@benjaminwarner.dev

How did we do it? First, we brought all the modern LLM architectural improvements to encoders, including alternating global & local attention, RoPE, and GeGLU layers, and added full model unpadding using Flash Attention for maximum performance (illustrated in the next post).

Benjamin Warner
@benjaminwarner.dev

Thanks. ModernBERT is a base model. It’ll need additional contrastive pretraining to really shine as a retrieval model, but our early results in the paper look promising. Hopefully there will be multiple open source retrieval tuned models to choose from early next year, including ColBERT finetunes.

Phil Nash
@philna.sh

That’d be sweet. I thought I saw a ColBERT file in the GitHub repo, so wondered what it was. But I get that it’ll need other training. Not my forte sadly, so I look forward to seeing those!