Valentina Pyatkin

🗓️ SwissText 2026 keynote speakers announced & registration open! We are delighted to welcome Prof. Dr. Alexandra Birch and Dr. Valentina Pyatkin as our keynote speakers. 📋 Register here: ema.uzh.ch/RHK4W Early-bird rates available throughout April, with additional student discounts. #NLProc 1/3

Excited to have the Big Picture workshop back for another iteration this year at @aclmeeting.bsky.social Submit your big picture ideas, consolidation work, phd thesis distillation, etc. by March 5th! www.bigpictureworkshop.com w/ Allyson Ettinger, @norakassner.bsky.social, @sebruder.bsky.social

🚨 New Study 🚨 @arxiv.bsky.social has recently decided to prohibit any 'position' paper from being submitted to its CS servers. Why? Because of the "AI slop", and allegedly higher ratios of LLM-generated content in review papers, compared to non-review papers.

Front Conference Zurich is coming up soon! On Friday, February 27, an amazing group of speakers will explore how AI is reshaping the way we work, from creativity and product design to engineering and collaboration 🤩 Our lineup: frontconference.com/schedule 🎟️ Your ticket: frontconference.com/tickets

We're at #NeurIPS2025 with papers, posters, workshops, fireside chats, & talks across the conference. Come learn about our latest research + see live demos!

Olmo 3 is notable as a "fully open" LLM - all of the training data is published, plus complete details on how the training process was run. I tried out the 32B thinking model and the 7B instruct models, + thoughts on why transparent training data is so important simonwillison.net/2025/Nov/22/...

Olmo 3 is out! 🤩 I am particularly excited about Olmo 3 models' precise instruction following abilities and their good generalization performance on IFBench! Lucky to have been a part of the Olmo journey for three iterations already.

Happy Halloween!

There’s plenty of evidence for political bias in LLMs, but very few evals reflect realistic LLM use cases — which is where bias actually matters. IssueBench, our attempt to fix this, is accepted at TACL, and I will be at #EMNLP2025 next week to talk about it! New results 🧵

I will be giving a talk at @eth-ai-center.bsky.social next week, on RLVR for verifiable instruction following, generalization, and reasoning! 📢 Join if you are in Zurich and interested in hearing about IFBench and our latest Olmo and Tülu works at @ai2.bsky.social

"Although I hate leafy vegetables, I prefer daxes to blickets." Can you tell if daxes are leafy vegetables? LM's can't seem to! 📷 We investigate if LMs capture these inferences from connectives when they cannot rely on world knowledge. New paper w/ Daniel, Will, @jessyjli.bsky.social

Hi #COLM2025! 🇨🇦 I will be presenting a talk on the importance of community-driven LLM evaluations based on an opinion abstract I wrote with Jo Kavishe, @victorojewale.bsky.social and @geomblog.bsky.social tomorrow at 9:30am in 524b for solar-colm.github.io Hope to see you there!

Now accepted to #neurips25 datasets & benchmarks! See you in San Diego! 🥳

🚀 Can open science beat closed AI? Tülu 3 makes a powerful case. In our new #WiAIRpodcast, we speak with Valentina Pyatkin (@valentinapy.bsky.social) of @ai2.bsky.social and the University of Washington about a fully open post-training recipe—models, data, code, evals, and infra. #WomenInAI 1/8🧵

"𝐋𝐋𝐌 𝐏𝐨𝐬𝐭-𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠: 𝐎𝐩𝐞𝐧 𝐒𝐜𝐢𝐞𝐧𝐜𝐞 𝐓𝐡𝐚𝐭 𝐏𝐨𝐰𝐞𝐫𝐬 𝐏𝐫𝐨𝐠𝐫𝐞𝐬𝐬 " 🎙️ On Sept 17, the #WiAIRpodcast speaks with @valentinapy.bsky.social (@ai2.bsky.social & University of Washington) about open science, post-training, mentorship, and visibility #WiAIR #NLProc

With fresh support of $75M from NSF and $77M from NVIDIA, we’re set to scale our open model ecosystem, bolster the infrastructure behind it, and fast‑track reproducible AI research to unlock the next wave of scientific discovery. 💡

On my way to Oxford: Looking forward to speaking at OxML 2025

I had a lot of fun contemplating about memorization questions at the @l2m2workshop.bsky.social panel yesterday together with Niloofar Mireshghallah and Reza Shokri, moderated by @pietrolesci.bsky.social who did a fantastic job! #ACL2025

I'll be at #ACL2025🇦🇹!! Would love to chat about all things pragmatics 🧠, redefining "helpfulness"🤔 and enabling better cross-cultural capabilities 🗺️ 🫶 Presenting our work on culturally offensive nonverbal gestures 👇 🕛Wed @ Poster Session 4 📍Hall 4/5, 11:00-12:30

🔥tokenization panel!

why is vancouver sushi so good? 🤤 (vancouver food in general actually)

This week is #ICML in Vancouver, and a number of our researchers are participating. Here's the full list of Ai2's conference engagements—we look forward to connecting with fellow attendees. 👋

In Geneva🇨🇭to attend the International Open-Source LLM Builders Summit and present OLMo and Tülu!

plus, some fun RL experiments

This new benchmark created by @valentinapy.bsky.social should be the new default replacing IFEval. Some of the best frontier models get <50% and it comes with separate training prompts so people don’t effectively train on test. Wild gap from o3 > Gemini 2.5 pro of like 30 points.

Introducing IFBench, a benchmark to measure how well AI models follow new, challenging, and diverse verifiable instructions. Top models like Gemini 2.5 Pro or Claude 4 Sonnet are only able to score up to 50%, presenting an open frontier for post-training. 🧵

Check out our take on Chain-of-Thought. I really like this paper as a survey on the current literature on what CoT is, but more importantly on what it's not. It also serves as a cautionary tale to the (apparently quite common) misuse of CoT as an interpretable method.

🚨Submission deadline extended to June 27th AoE!🚨 Our reviewer interest form is also open! See below for more details👇

I collected some folk knowledge for RL and stuck them in my lecture slides a couple weeks back: web.mit.edu/6.7920/www/l... See Appendix B... sorry, I know, appendix of a lecture slide deck is not the best for discovery. Suggestions very welcome.

i created a gist with some non default llm courses gist.github.com/yoavg/95bbc5...

I’m thrilled to share RewardBench 2 📊— We created a new multi-domain reward model evaluation that is substantially harder than RewardBench, we trained and released 70 reward models, and we gained insights about reward modeling benchmarks and downstream performance!

RewardBench 2 is here! We took a long time to learn from our first reward model evaluation tool to make one that is substantially harder and more correlated with both downstream RLHF and inference-time scaling.

Happy to announce the first workshop on Pragmatic Reasoning in Language Models — PragLM @ COLM 2025! 🎉 How do LLMs engage in pragmatic reasoning, and what core pragmatic capacities remain beyond their reach? 🌐 sites.google.com/berkeley.edu/praglm/ 📅 Submit by June 23rd

Our discussion period just started. Authors, please read our instructions carefully. We require responses by June 2. But, what you really want to hear about is stats .... right? -> 🧵

Is very classic that most people don't know the Tulu 3 paper coined the term RLVR