Itay Itzhak @ COLM 🍁

We vibe-tested our own paper, but the reviewers gave us the actual metrics: "From Feelings to Metrics" has been accepted to #COLM2026! 🎉 We turn "this model just feels better" into structured, user-aware evaluation! #COLM people - let’s grab a coffee and vibe-test some research in person! ☕️🌴

In Rio for #ICLR2026 🇧🇷 and already had my first açaí! 🍧 Come chat LLM safety and evaluation, and stop by our ManagerBench poster (w/ @adisimhi)! - Tomorrow (Friday) @ 10:30 - Poster Session 3, Pavilion 4

Introducing Global PIQA, a new multilingual benchmark for 100+ languages. This benchmark is the outcome of this year’s MRL shared task, in collaboration with 300+ researchers from 65 countries. This dataset evaluates physical commonsense reasoning in culturally relevant contexts.

Had a blast at CoLM! It really was as good as everyone says, congrats to the organizers 🎉 This week I’ll be in New York giving talks at NYU, Yale, and Cornell Tech. If you’re around and want to chat about LLM behavior, safety, interpretability, or just say hi - DM me!

Thrilled to be part of this work led by @adisimhi.bsky.social ! ManagerBench reveals a critical problem: ✅ LLMs can recognize harm ❌ But often choose it anyway to meet goals 🤖 Or overcorrect and become ineffective We need better balance! A must-read for safety folks!

Traveling to #COLM2025 this week, and here's some work from our group and collaborators: Cognitive biases, hidden knowledge, CoT faithfulness, model editing, and LM4Science See the thread for details and reach out if you'd like to discuss more!

At #ACL2025 and not sure what to do next? GEM 💎² is the place to be for awesome talks on the future of LLM evaluation. Come hear @GabiStanovsky, @EliyaHabba, @LChoshen and others rethink what it means to actually evaluate LLMs beyond accuracy and vibes. Thursday @ Hall C!

In Vienna for #ACL2025, and already had my first (vegan) Austrian sausage! Now hungry for discussing: – LLMs behavior – Interpretability – Biases & Hallucinations – Why eval is so hard (but so fun) Come say hi if that’s your vibe too!

Excited to share our paper: "Chain-of-Thought Is Not Explainability"! We unpack a critical misconception in AI: models explaining their steps (CoT) aren't necessarily revealing their true reasoning. Spoiler: the transparency can be an illusion. (1/9) 🧵

Are you recovering from your @colmweb.org abstract submission? GEM has a non-archival track that allows you to submit a two-page abstract in parallel? Our workshop deadline is soon, please consider submitting your evaluation paper! You can find our call for papers at gem-benchmark.com/workshop

New paper alert! Curious how small prompt tweaks impact LLM accuracy but don’t want to run endless inferences? We got you. Meet DOVE - a dataset built to uncover these sensitivities. Use DOVE for your analysis or contribute samples -we're growing and welcome you aboard!

1/13 LLM circuits tell us where the computation happens inside the model—but the computation varies by token position, a key detail often ignored! We propose a method to automatically find position-aware circuits, improving faithfulness while keeping circuits compact. 🧵👇

🚨🚨 New preprint 🚨🚨 Ever wonder whether verbalized CoTs correspond to the internal reasoning process of the model? We propose a novel parametric faithfulness approach, which erases information contained in CoT steps from the model parameters to assess CoT faithfulness. arxiv.org/abs/2502.14829

We usually blame hallucinations on uncertainty or missing knowledge. But what if I told you that LLMs hallucinate even when they *know* the correct answer - and they do it with *high certainty* 🤯? Check out our new paper that challenges assumptions on AI trustworthiness! 🧵👇

GEM is so back! Our workshop for Generation, Evaluation, and Metrics is coming to an ACL near you. Evaluation in the world of GenAI is more important than ever, so please consider submitting your amazing work. CfP can be found at gem-benchmark.com/workshop