Post

Jeremy Howard
@howard.fm

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

Jeremy Howard
@howard.fm

We created ModernBERT because encoder-only architectures are ideal for the kinds of real-world problems that come up every day, like retrieval (e.g. for RAG), classification (such as content moderation), & entity extraction. But they're under-appreciated, with little investment.

4 reposts68 likes

Jeremy Howard
@howard.fm

6 years after BERT, we have a replacement: ModernBERT! @answerdotai, @LightOnIO (et al) took dozens of advances from recent years of work on LLMs, and applied them to a BERT-style model, including updates to the architecture and the training process, eg alternating attention.

Jeremy Howard
@howard.fm

Fancy GenAI stuff like GPT 4 is too big, slow, private, and expensive for many jobs. Consider that the original GPT-1 was 117m params. Llama 3.1, by contrast, has up to 405 billion params! 😲 These models are slow, expensive, and *not yours to control*.

Jeremy Howard
@howard.fm

The popular buzz around GenAI has obscured the role of encoder-only models. These are the workhorses of practical language processing, the models that are actually being used for such workloads right now in many scientific and commercial applications.

Jeremy Howard
@howard.fm

GPT & friends are hamstrung: as generative models, they are mathematically “not allowed” to “peek” at later tokens. They can only ever look *backwards*. This is in contrast to encoder-only models like BERT, which are trained so each token can look forwards *and* backwards.