Lukas Gienapp

Super happy that our paper “Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs” just won Best Student Paper at #SIGIR2026🏆 It argues against the reflex to reach for an LLM whenever you need relevance judgments. What we found 🧵:

Given the many problems that LLM-as-a-judge brings, this opens up new possibilities for scalable, reliable evaluation; and, continuing the thread from SIGIR 2025, this again confirms: human judgment stays the gold standard for evaluating IR and RAG. It's just more efficient to evaluate with it now 😉

Super happy that our paper “Topic-Specific Classifiers are Better Relevance Judges than Prompted LLMs” just won Best Student Paper at #SIGIR2026🏆 It argues against the reflex to reach for an LLM whenever you need relevance judgments. What we found 🧵:

Two papers accepted at SIGIR'26 in Melbourne! Both tackle the same question: how do we scale IR evaluation reliably without compromising on human judgment as the gold standard? Spoiler: LLMs-as-a-judge does not work. 🧵1/5

🧵4/5 Our second paper provides supporting evidence from the RAG side: when we replace LLM-generated ground truth answers with human-written ones, reference-based evaluation scores shift significantly. Humans and LLM judges show low agreement, highlighting the importance of human grounding.

Two papers accepted at SIGIR'26 in Melbourne! Both tackle the same question: how do we scale IR evaluation reliably without compromising on human judgment as the gold standard? Spoiler: LLMs-as-a-judge does not work. 🧵1/5

🧵2/5 Our first paper proposes a new solution to the classic unjudged document problem: topic-specific judge adapters. We propose finetuning a small ranking model with LoRA on a single assessor's judgments for a single topic, deliberately overfitting to that assessor's notion of relevance.