Kyle Lo

I'm at #colm2026, Mon-Thurs Supporting two papers from @ai2.bsky.social days 🐟Olmo Hybrid, 7B hybrid model 🐠Cracks in the Foundation, long context recipes Hoping to have fun chats w folks about scaling LM data & evals, ☺️ unbiased on pretrain, posttrain, all the 🚂s

excited to see frens at #icml2026 & present 🐟 Olmix: efficient data mixing under token constraints & evolving data domains 🐡 How2Everything: mining the web for diverse procedural tasks for train & eval 🐠 happy to chat data & evals, both pre & post-training

Olmix blog: allenai.org/blog/olmix Extended paper: arxiv.org/abs/2602.12237

our new Olmo Hybrid model combines attention with linear RNN layers 🍣training efficiency is crazy good. the model reaches same MMLU score as Olmo 3 in 50% of the tokens. also see this in many other tasks as always: weights, data, ckpts, training code, etc. all fully open

our paper on data mixing for LMs is out! while building Olmo 3, we saw gaps between data mixing literature and real practice 🐠choosing proxy size, # runs, sampling, regression, constraints.. 🐟data shifts during LM dev: can we reuse past experiments? Olmix tackles them all!

one of my favorite topics is dealing with data constraints! what if your proposed mix is 30% code but you don't have enough code? we can repeat our data until we hit target proportions, but too much is risky we view data mixing as (data) constrained optimization

incredibly fun project led by our intern yapei chang we mined the web for thousands of real-world “how to do X” step by step instructions and turned it into a dataset, synth data training procedure, eval suite, etc.

some thoughts about skill degradation w/ AI coding im onboard w views that "english is the new programming language" & "software engineering", translating ambiguous goals to technical specs/execution, is still a skill. im more concerned w shift from my role as a writer to a reviewer and

i dont feel worse at this even if im not writing papers from-scratch as much as during early career but coding feels different due to mismatch between what i express to the system (english) and what the system returns (code). i've already realized some gaps in libraries I used to know well.

some thoughts about skill degradation w/ AI coding im onboard w views that "english is the new programming language" & "software engineering", translating ambiguous goals to technical specs/execution, is still a skill. im more concerned w shift from my role as a writer to a reviewer and

during neurips, we kept the RL run going & model kept getting better 😂 Olmo 3.1 is a.. 🐡 32B Thinking, still best fully-open model to-date 🐠 32B Instruct, for ppl who hate long yapping, as good as qwen3 we added 10 more pages to the paper! thx for community feedback from convos at neurips

paper has: 🐟 more on our eval ideology 🦈 more baselines 🍣 more about RL Zero etc we picked final model (internally called moonlit surfer 🌛🏄) not just on bench scores but good vibes 🥰

I'll be at #NeurIPS2025 from Tues-Sat! Come say hi 👋 if you wanna chat about 🦈 olmo 3 stories 🐟 pretraining data & evals 🍣 midtraining shouldnt exist 🐠 model specialization 🐡 AI for education 🍥 tabletop games

we released Olmo 3! lot of exciting stuff but wanna focus on: 🐟Olmo 3 32B Base, the best fully-open base model to-date, near Qwen 2.5 & Gemma 3 on diverse evals 🐠Olmo 3 32B Think, first fully-open reasoning model approaching Qwen 3 levels 🐡12 training datasets corresp to different staged training

🍏Try the model: playground.allenai.org 🍍Download the collection: huggingface.co/collections/... 🍌Read the blog: allenai.org/blog/olmo3 🍎And our 100+ page paper lol www.datocms-assets.com/64837/176364...

we released Olmo 3! lot of exciting stuff but wanna focus on: 🐟Olmo 3 32B Base, the best fully-open base model to-date, near Qwen 2.5 & Gemma 3 on diverse evals 🐠Olmo 3 32B Think, first fully-open reasoning model approaching Qwen 3 levels 🐡12 training datasets corresp to different staged training

🍨Data quality signals matter but also how you use them! Traditional ways of using data quality is to threshold: Define a cutoff and take all the documents above that threshold. But why not sample *proportional* to data quality? We use Quality-Aware Upsampling to do exactly this

we released Olmo 3! lot of exciting stuff but wanna focus on: 🐟Olmo 3 32B Base, the best fully-open base model to-date, near Qwen 2.5 & Gemma 3 on diverse evals 🐠Olmo 3 32B Think, first fully-open reasoning model approaching Qwen 3 levels 🐡12 training datasets corresp to different staged training

🦈Invest in your experimental design! We create evals better suited for different compute scales, with our "easy" set of tasks+metrics able to support very small scale experiments before switching to our "main" set of evals, on which smaller models are below noise floor

why intern at Ai2? 🐟interns own major parts of our model development, sometimes even leading whole projects 🐡we're committed to open science & actively help our interns publish their work reach out if u wanna build open language models together 🤝 links 👇