Post

Kyle Lo
@kylelo.bsky.social

we released Olmo 3! lot of exciting stuff but wanna focus on: 🐟Olmo 3 32B Base, the best fully-open base model to-date, near Qwen 2.5 & Gemma 3 on diverse evals 🐠Olmo 3 32B Think, first fully-open reasoning model approaching Qwen 3 levels 🐡12 training datasets corresp to different staged training

Kyle Lo
@kylelo.bsky.social

🦈Invest in your experimental design! We create evals better suited for different compute scales, with our "easy" set of tasks+metrics able to support very small scale experiments before switching to our "main" set of evals, on which smaller models are below noise floor

Kyle Lo
@kylelo.bsky.social

🍣Data mixing is a little too powerful It's easy to learn "optimal" mixes that oversample from certain pockets heavily. eg, STEM docs are valuable for climbing MMLU & but you don't have infinite STEM docs We approach mixing as Token Constrained Optimization over diverse evals

Kyle Lo
@kylelo.bsky.social

🍨Data quality signals matter but also how you use them! Traditional ways of using data quality is to threshold: Define a cutoff and take all the documents above that threshold. But why not sample *proportional* to data quality? We use Quality-Aware Upsampling to do exactly this

0 reposts3 likes

Kyle Lo
@kylelo.bsky.social

🍕Finally, we all know midtraining is an exciting time to get a ton of performance boost But team organization to sustain consistent model improvements (without burnout) is important! We have explorers "own" target capabilities & centralized assessment team run "integration tests"