Ai2 (@ai2.bsky.social)

27 Aug

AI’s biggest role in science may not be answering questions. It may be helping scientists find which questions are worth asking. At Providence Swedish, AutoDiscovery surfaced an unexpected immune signal in a heavily studied cancer dataset—and follow-up research confirmed it. 🧵 buff.ly/LTacUQl

LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it? Our new eval reveals even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them. 🧵 buff.ly/WjOSxcx

Prompt changes that helped one judge could hurt another, & more reasoning or access to related papers didn't consistently improve results. And 2 purpose-built novelty judges performed worse than a simple prompted Claude Opus 4.6 baseline.

buff.lyGitHub - allenai/novelty-eval: Benchmark and controlled study of LLM-based novelty judges for research ideas. Code and data for "Old Ideas, Novel Problems" (arXiv 2610.02022)Benchmark and controlled study of LLM-based novelty judges for research ideas. Code and data for "Old Ideas, Novel Problems" (arXiv 2610.02022) - allenai/novelty-eval

We hope our benchmark helps researchers build more reliable novelty judges, while our reusable pipeline makes it easier to create new eval sets as AI-generated research advances. 💻 Blog: buff.ly/PRpiYGP 📊 Data: buff.ly/XeNZgd5 📄 Paper: buff.ly/14yas5t

We evaled judge models on 2 sets: 👤👤 Human-only pairs (stronger vs. weaker evidence of novelty) 👤🤖 Human ideas w/ novelty evidence paired w/ basic LLM ideas, assumed less novel. Judges either picked the more novel idea in each pair or labeled single ideas as novel or not.

On human–AI pairs, GPT-5.4 chose the human idea 93.5% of the time when asked which idea reviewers had judged novel. Asked simply which idea was novel, that share fell to 40.9%. Prompt wording had a smaller effect on which idea models chose in human-only pairs.

At Ai2, we're building tools like AutoDiscovery to uncover unexpected patterns in data & Theorizer to propose candidate scientific laws from papers. As AI tools like these advance, we need reliable ways to tell whether the ideas they generate are genuinely new.

To build our benchmark, we mined ICLR 2026 reviews for explicit comments on originality. We kept papers that reviewers agreed were (or were not) original & whose scores were unusually high or low within their research area, giving us examples of more and less novel work.

We're at #EMNLP2026 with papers & talks across the conference. Come say hello and learn about our latest research!

Developing a strong research idea means knowing what prior work establishes, where gaps remain, and how your contribution addresses them. Introducing LitPivot, which helps refine early ideas by surfacing relevant literature as they evolve. buff.ly/iYol1M4

In a study with 17 researchers, participants using LitPivot selected significantly more unique papers than when using a literature Q&A chatbot, reported stronger understanding of the literature, and produced ideas experts rated as significantly better grounded.

buff.lyLitPivot: Developing research ideas through dynamic contextualization within the literature landscape | Ai2LitPivot helps researchers refine early research ideas through literature-supported critiques and revisions as their proposals evolve.

LitPivot builds on our broader work on AI for science, moving from helping researchers find and synthesize the literature toward helping them use it to challenge, refine, & develop new ideas. Blog: buff.ly/Cd39NYj

LitPivot breaks an idea into three facets: 🔎 Problem 💡 Contribution 🧪 Evaluation Select one, and the system organizes related papers around that facet, clusters them by shared themes, & highlights the literature most relevant to what you’re refining.

LitPivot then checks a selected facet against papers you choose, surfacing overlaps, gaps, & limitations in prior work + suggesting literature-informed revisions. When one facet changes, LitPivot also flags others that may be out of sync.

We investigate “literature-initiated pivots,” where research ideas and the literature they build on evolve together. Prior work can reveal that a contribution isn’t novel or an evaluation should change. As ideas change, so does the most relevant literature.

OlmoEarth v1.3 is out. 🌍 State-of-the-art embeddings from the same foundation model family, precomputed for every 10 m pixel on Earth. We’re excited to give you more ways to work with OlmoEarth, including a new browser tool to explore and customize maps. Full rundown in 🧵

Global embeddings are expensive to produce, so we computed them for the entire planet and shared them on @source.coop, where everyone can access the benefits. A land-cover map or a change analysis becomes a small classifier you train on top.

buff.lyOlmoEarth v1.3: State-of-the-art embeddings for the whole planet | Ai2OlmoEarth v1.3 brings state-of-the-art satellite embeddings, precomputed global data, and a browser-based Embeddings Explorer for mapping and analyzing the planet.

Everything is in this release. The v1.3 Base model, global embeddings, the Embeddings Explorer, the tech report, PLANTEUR (a new crop-type benchmark for the tropics), and the interactive story. All open. Start with the blog: buff.ly/u0tDGAF

OlmoEarth’s 128-dim embeddings lead AlphaEarth Foundations & Tessera v2 on 10 of 11 evals. The 64-dim version, at half the storage, leads on 7. On the ground, a few dozen labeled points are enough for a usable first map of smallholder crops in Kenya or fire damage in Indonesia.

New to embeddings? Our interactive story walks you through what they are. A volcanic eruption, mangroves mapped in a few taps, a forest cleared in a week, and more. Each stop shows what the embeddings reveal and how they could help you.

Explore the embeddings, then ask them questions. What changed here? What else looks like this? The OlmoEarth Embeddings Explorer puts that in a browser. Flip between years, find similar places, or drop a few labels and train a classifier on the spot. No GPU, no sign-in.

Today we’re introducing FairStress, a benchmark for testing demographic fairness beyond a model’s final answer: how readily it changes a well-supported decision & how it explains its choice. 🧵 buff.ly/lWsW77r

Since LLMs have various biases in their decisions and reasoning, we then examined how training data might influence this, & find that LLMs are never really taught fair decision-making yet are often aligned to prevent discrimination—which may explain over-correction patterns.

buff.lyFairStress: A deeper evaluation of fairness in LLM decision-making | Ai2FairStress is a new benchmark that reveals demographic differences in how LLMs defend and explain their decisions, even when their final choices appear fair.

We hope FairStress gives researchers a way to uncover hidden asymmetries in model decision-making, test fixes, & develop more consistent systems. 📝 Blog: buff.ly/McL5dhS 📄 Paper: buff.ly/jpbutOP

FairStress also reveals differences in how models explain their choices. Across 11 open-weight models, minority-coded choices used somewhat more positive & identity-related language, while majority-coded choices leaned more on terms like “evidence” & “criteria.”

We tested 9 approaches for reducing these gaps. Most didn’t reliably improve symmetric robustness; some hurt performance. Older benchmarks like BBQ can miss overcorrection, but FairStress reveals the hidden asymmetries they overlook.

FairStress reveals asymmetric unfairness in LLMs In a hiring prompt, for example, the eval might change “the candidate is a woman” to “the candidate is a man” while keeping qualifications identical, then probe whether a model becomes easier to sway from its original decision.

We found that models were more likely to abandon the candidate with stronger experience & qualifications when that candidate was majority-coded – for example, described as white in a white/Black comparison – than minority-coded, such as Black in the same comparison.

Models are increasingly being used to support decisions about people—from hiring & lending to education & healthcare. That makes it important to test not only what decisions models make, but whether they apply the same standards across demographic groups.

What if you could simulate global extreme weather at 3 km resolution for ~1/300th the cost? Our new HiRO-ACEv2 system does just that, while matching X-SHiELD’s statistics to within 3% even for events as rare as 1 in a million data points. 🧵 buff.ly/mZnScdv

Together, ACE + HiROv2 can generate years of global climate data at resolutions fine enough to capture storms, severe winds, & other local extreme events. And it’s cheap: global 3 km simulations cost ~300× less than X-SHiELD, with even bigger savings for individual regions.

buff.lyHiRO-ACEv2: Simulating extreme weather at 3 km resolution for a fraction of the cost | Ai2HiRO-ACEv2 captures extreme rain, wind, and pressure events at 3 km resolution for about 1/300th the cost of X-SHiELD, making detailed global climate simulations more accessible to researchers.

Next, we want HiRO-ACEv2 to work across more climates—not just the one it was trained on. We’re training it on observed-weather & +4°C datasets, adding outputs like temperature, snow, & ice + exploring ways to make HiROv2 10×+ faster. Read more: buff.ly/NAfRlQI

HiRO-ACEv2 builds on HiRO-ACE by downscaling near-surface winds & sea level pressure in addition to rainfall. It works in 2 stages: 🌍 ACE, an AI climate emulator, simulates climate at a coarser 100 km resolution 🔎 HiROv2, an AI downscaling model, sharpens the outputs to 3 km

HiROv2 is especially strong on extremes: it matches the distribution of extreme events in X-SHiELD to within 3% for 1-in-a-million rain, wind, & pressure predictions. It also captures cyclone energy, realistic hurricanes, & strong downslope winds. Demo: ace.allen.ai

Kai Van Brunt adapted Molmo 2, our open model for video understanding, to a domain far outside its training data: sonar video of salmon. Meet Molmo2Fish, a prototype that tracks fish + can correct some of its mistakes through natural-language guidance. 🧵 buff.ly/fRNkIGY

Kai fine-tuned Molmo 2 on sonar video then trained it on examples of tracking errors, instructions, & corrections. In tests, Molmo2Fish could fix some mistakes after guidance—for example, when two salmon had been assigned the same track.

Molmo 2 already knew how to track objects in video. Its open code let Kai modify the model, while its open training data helped him understand what it had learned and prepare his own data—turning a general-purpose model into one tailored to his research.

Tracking salmon by analyzing video helps researchers monitor their populations. But when a salmon-tracking system makes a mistake, like an overcount, fixing it can mean editing clips frame after frame by hand. What if you could just tell the system to fix what it got wrong?

We tested frontier models including Astra & Opus 5.5 on TutorMoments, our eval for whether LLMs help students solve problems themselves or do too much for them. In general, models scored better with detailed prompts than when told only to “tutor well.” buff.ly/GpAdLkA

When asked to “tutor well,” Claude Sonnet 5.5 (low effort) was best at asking students to work through more of a math problem themselves when teachers recommended it. With explicit tutoring instructions, GPT-6 Sol (no reasoning) outscored Claude Opus 4.8 overall.

Compare responses from models & human tutors in the new explorer experience, where you can also examine model tutoring performance alongside response time and cost: buff.ly/GpAdLkA

To develop TutorMoments, we had experienced teachers mark moments in math tutoring transcripts when tutors should offer support or ask more of a student. We asked models to continue the conversations from those moments + scored their replies against the teachers’ judgments.

In these tests, GPT-6 models over-help less than earlier OpenAI models. They also score higher overall when we explicitly instruct them to offer help where needed while leaving students to work through parts of a problem they can handle themselves.

luma.comOpen Together: AI Builders Unite · LumaCalling all AI Builders to San Francisco. Let's come together. Show strength in numbers. Celebrate open models. And dance for AI freedom. Friday October 16 is…

This Friday, meet the open source AI community at Open Together, co-hosted with @hf.co. Join us at The Midway in SF on Oct 16 — bringing the community together to help build a future where AI is open, secure, and fully transparent. RSVP: luma.com/OpenTogether

allenai.orgAi2 at COLM 2026: Open research, from models to agents for science | Ai2Ai2 joined COLM 2026 to share advances in open language models, training infrastructure, and AI for science, including Olmo Hybrid, Asta, and Bolmo, while connecting with the broader research communit...

That's a wrap on #COLM2026! Thank you to everyone who stopped by our booth, joined our evening gathering, & talked open research with us. Catch up on the papers, interviews, & highlights from the week. 👇 allenai.org/blog/colm-20...

We’re sharing the engineering work behind Ai2’s GPU scheduler—our new approach to allocating compute across research teams. On our largest H100 cluster, median queue wait fell from 5 minutes to 24 seconds. 🧵 buff.ly/4UsJEYe

Over a 30-day test with our new scheduler, teams received 98% of the GPU hours they were owed, accounting for actual demand. Cluster occupancy stayed at 98%, with spare capacity available to interruptible work that didn’t draw down a team’s budget.

Read more about the design decisions behind our scheduler, rollout lessons, & what we’re tackling next: buff.ly/4UsJEYe

Managers now assign GPU-time budgets to research programs & projects. Our fair-share scheduler compares recent use with those budgets, moving work from underused allocations ahead in-queue.