Post

This week we released ModernBERT, the first encoder to reach SOTA on most common benchmarks across language understanding, retrieval, and code, while running twice as fast as DeBERTaV3 on short context and three times faster than NomicBERT & GTE on long context.

Phil Nash

@philna.sh

24 Dec 2024

This is awesome! Are there any docs on how to use ModernBERT for ColBERT?

Benjamin Warner

@benjaminwarner.dev

Thanks. ModernBERT is a base model. It’ll need additional contrastive pretraining to really shine as a retrieval model, but our early results in the paper look promising. Hopefully there will be multiple open source retrieval tuned models to choose from early next year, including ColBERT finetunes.

24/12/24, 7:26 am0 reposts2 likes

Carsten Schnober

@carschno.bsky.social

27 Dec 2024

Can you share the model fine-tuned on MS-MARCO? Also, would you elaborate on how the token embeddings are aggregated? Mean pooling as in SentenceBert?

Phil Nash

@philna.sh

24 Dec 2024

That’d be sweet. I thought I saw a ColBERT file in the GitHub repo, so wondered what it was. But I get that it’ll need other training. Not my forte sadly, so I look forward to seeing those!

Benjamin Warner

@benjaminwarner.dev

24 Dec 2024

We evaluated ModernBERT on MLDR using ColBERT-style retrieval using that code. That process was smaller scale than a full ColBERT finetune, which would need additional contrastive training, likely use multiple teacher models, etc as detailed here by @bclavie.bsky.social www.answer.ai/posts/2024-0...

deen

@sir-deenicus.bsky.social

24 Dec 2024

Hi, are there any examples or docs on how to use for entailment/NLI--is modernBERT suitable for 0-shot classification?

Benjamin Warner

@benjaminwarner.dev

24 Dec 2024

ModernBERT is a “foundation model” so you’ll either need to finetune it for entailment/NLI or wait for someone else to finetune it. I suspect it would be good at NLI once finetuned.