Post

🏎️ I've never seen MoE models for embedding & search this small: 1.5M, 27M & 115M parameters! VAGO Solutions' MoEdernColBERT models take on much larger dense retrievers, with 115M near or ahead of Qwen3-Embedding-0.6B in their overview at 8× the encoding throughput. Details 🧵

All three are Apache 2.0 multilingual multi-vector retrievers for German, English, Spanish, Dutch, French, Italian & Portuguese. And yes, 1.5M is the total parameter count, with just 0.88M active per token. Crazy! huggingface.co/collections/...

You can use all three with Sentence Transformers' MultiVectorEncoder, with trust_remote_code=True. Here, 1.5M matches the German mountain query and English capital query to the right documents. Higher MaxSim scores mean a better match.

In their overview, 115M beats Qwen3-Embedding-0.6B on 9 of 10 quality rows. English NanoBEIR is the exception: 0.643 vs. 0.648, while the six-language average is 0.585 vs. 0.574. Non-English NanoBEIR uses their translations. huggingface.co/VAGOsolution...

Even 1.5M edges out IBM's granite-97m-multilingual-r2 on English NanoBEIR: 0.591 vs. 0.589 nDCG@10 in their evaluation. The gap is much larger on GermanQuAD: 0.976 vs. 0.918. A 1.5M model holding its own against a 97M encoder is pretty nuts.

At 27M, it already passes Qwen3-Embedding-0.6B on their cross-lingual MKQA average: 0.684 vs. 0.681 Recall@20. Queries in six languages search English Wikipedia, at ~22× fewer parameters and 47× the reported encoding throughput. huggingface.co/VAGOsolution...

Each transformer layer routes a token to 2 of 8 experts, plus one shared expert. They also shrink the embedding table by giving each token a short vector that's projected to the hidden size. For 1.5M, that saves ~3.4M parameters, although it doesn't result in speedups.

A good part of the quality gap may come from multi-vector retrieval: each token keeps a 128d vector, and MaxSim lets query tokens match different document tokens. That means more vectors to store & score than one dense vector per text, even with such a tiny encoder.