Post

Webis Group
@webis.de

Our paper on self-distillation for training bi-encoders got accepted at #ICTIR2025! By exploiting pretrained encoder capabilities, our approach eliminates expensive teacher models and batch sampling while maintaining the same effectiveness.

Webis Group
@webis.de

The key idea: we can use the similarity predicted by the encoder itself between positive and negative documents to scale a traditional margin loss. This performs implicit hard negative mining and is hyperparameter-free.

Webis Group
@webis.de

Results on BEIR demonstrate that our method matches teacher distillation effectiveness, while using only 13.5% of the data and achieving 3-15x training speedup. This makes effective bi-encoder training more accessible, especially for low-resource settings.

0 reposts1 like

Webis Group
@webis.de

Congratulations to the authors @lgnp.bsky.social @deckersniklas.bsky.social @martin-potthast.com @hscells.bsky.social ! 📄 Preprint: arxiv.org/abs/2407.21515 💻 Code: github.com/webis-de/ada...

arxiv.orgLearning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance MarginsRepresentation-based retrieval models, so-called biencoders, estimate the relevance of a document to a query by calculating the similarity of their respective embeddings. Current state-of-the-art bien...