Post

How does this compare to openai's embedding model? We don't have an immediate need to move off openai, but would love the flexibility ... without regressions

We haven't compared it directly, although generally OpenAI's embeddings seem to trail the better open models, so I expect it would be an improvement.

Great! Looking forward to trying this out sometime. Since we're an AWS shop we'll probably consolidate on Bedrock, and although it has an embedding service, I'd love to use that opportunity to pick something that's actually best for our needs.

ModernBERT is available as a slot-in replacement for any BERT-like model, with both 139M param and 395M param sizes. It has a 8192 sequence length, is extremely efficient, is uniquely great at analyzing code, and much more. Read this for details: huggingface.co/blog/modern...

Seven months ago, @bclavie.bsky.social kicked things off, and soon Benjamin Warner & @nohtow.bsky.social joined him as project co-leads. I don't think anyone quite knew what we were getting in to… It turns out that training a new, SoTA model from scratch is actually pretty hard. Who knew? 🤷

We created ModernBERT because encoder-only architectures are ideal for the kinds of real-world problems that come up every day, like retrieval (e.g. for RAG), classification (such as content moderation), & entity extraction. But they're under-appreciated, with little investment.

Very cool! Any plans to come up with a multilingual version?

This sounds very interesting

Ooh! This looks very promising. I’ve been thinking of replacing some classification steps in a workflow with local models. I’m curious how fast they can be on CPU only hardware(compared to slow sonnet api calls).

Yes, this was my thought exactly. There's tons of stuff you want a little bit language model smarts for but you don't need need a full chatbot. This could be great for that, especially if it can run in the web server directly.

This could be very useful for my team at our healthcare company. Thanks for your work! I see brief mention of the dataset in the paper. Can you provide more info on it? Is it proprietary? A conglomerate of a bunch of open source datasets?

I'd like to know how does it compare with T5? P.S.: I'm the author of Wikipedia BERT and T5 and basically all the Transformer-related pages. If you have comments about how the pages can be improved I can do it. (Yes ULMFit had already been on the BERT page)

T5 is an Encoder-Decoder model -- a different part of the family tree. It's primarily a generative model, whereas ModernBERT isn't, so it doesn't make that much sense to compare them.

Yi Tay etc disagree: www.yitay.net/blog/model-a... 2024-07-16 > BERT-style models pretty much deprecated at this point because there is a strictly better alternative... if the decoder was getting in the way... yanking out the encoder performed just as competitive as a BERT encoder.