Post

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

ModernBERT is available as a slot-in replacement for any BERT-like model, with both 139M param and 395M param sizes. It has a 8192 sequence length, is extremely efficient, is uniquely great at analyzing code, and much more. Read this for details: huggingface.co/blog/modern...

Seven months ago, @bclavie.bsky.social kicked things off, and soon Benjamin Warner & @nohtow.bsky.social joined him as project co-leads. I don't think anyone quite knew what we were getting in to… It turns out that training a new, SoTA model from scratch is actually pretty hard. Who knew? 🤷

We created ModernBERT because encoder-only architectures are ideal for the kinds of real-world problems that come up every day, like retrieval (e.g. for RAG), classification (such as content moderation), & entity extraction. But they're under-appreciated, with little investment.

Fancy GenAI stuff like GPT 4 is too big, slow, private, and expensive for many jobs. Consider that the original GPT-1 was 117m params. Llama 3.1, by contrast, has up to 405 billion params! 😲 These models are slow, expensive, and *not yours to control*.

The popular buzz around GenAI has obscured the role of encoder-only models. These are the workhorses of practical language processing, the models that are actually being used for such workloads right now in many scientific and commercial applications.

GPT & friends are hamstrung: as generative models, they are mathematically “not allowed” to “peek” at later tokens. They can only ever look *backwards*. This is in contrast to encoder-only models like BERT, which are trained so each token can look forwards *and* backwards.