Epoch AI

We are publishing 4 new benchmark reviews: 1 is Verified, and 3 are Flawed. 4 more benchmarks lacked sufficient information to assign a verdict. Our main bottleneck to reviewing more benchmarks is reviewer capacity. Info on applying to join our team is in the thread!

Not Enough Info: we couldn’t access enough to assign a verdict. Developers get a week to respond before we apply this label.

We are excited to keep reviewing benchmarks, and if you made it this far, you probably are too! Apply now if you’re interested in helping raise benchmarking standards: jobs.lever.co/epoch-ai/de...

Verified: the benchmark can broadly be interpreted as described by the developer, and any errors don’t substantially affect results. We publish a full review, including weaknesses and limitations.

epoch.aiBenchmark Reviews documentation – Included benchmarksDocumentation for Epoch AI’s Benchmark Reviews: the rubric we use to review external AI benchmarks, what the Verified, Flawed and Not enough information verdicts mean, how we choose which benchmarks to review, and answers to frequently asked questions.

Flawed: the benchmark has one or more substantive flaws users need to know about to interpret results, most often >20% of tasks with accuracy-impacting errors. We publish a limited write-up of the flaws.

Benchmarks assess AI capabilities, but their quality varies substantially. We hope to be a consistent source of information on benchmark quality. See all of our reviews: epoch.ai/data/benchm...

epoch.aiBenchmark Reviews documentation – MethodologyDocumentation for Epoch AI’s Benchmark Reviews: the rubric we use to review external AI benchmarks, what the Verified, Flawed and Not enough information verdicts mean, how we choose which benchmarks to review, and answers to frequently asked questions.

Reviews assign each benchmark a verdict based on our rubric: epoch.ai/data/benchm...

We recorded over a thousand runs of Slay the Spire and its sequel for our recent benchmark, SpireClimb. Over the coming weeks, we'll publish them all on our YouTube channel, complete with subtitles showing the models' in-game thinking.

youtube.comSpireClimb video archiveAn archive of videos showing LLMs playing Slay the Spire. We measured their win rates to produce SpireClimb, our Slay the Spire benchmark.

And the playlist where we're uploading all videos: www.youtube.com/playlist?li... We don't plan to update the playlist with videos from future models' runs on SpireClimb, but we hope this archive offers interesting insights into how current models play the game.

youtube.comastra sts2 a10 defect seed3 winEnjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.

Rather than being recorded live, the videos are reconstructed from the models’ play data (inputs and thinking). More on the SpireClimb benchmark:

Epoch AI@epochai.bsky.social

Six months ago, frontier AI models could barely beat Slay the Spire. Now, according to our new SpireClimb benchmark, they win 90%+ of runs at the easiest difficulty, and up to 20% of runs at the hardest.

Check out the run that produced the clip: www.youtube.com/watch?v=Ifx...

How far behind the closed frontier are open-weight cyber capabilities? About 6–7 months. Zai’s GLM-5.3 is the highest-scoring open-weight model on our Cyber ECI. Its score of 160 beats OpenAI’s GPT-5.4, released in March.

epoch.aiOpen-weight AI models are 6-7 months behind on cyberZ.ai’s GLM-5.3 is the top open-weight model on Epoch’s Cyber ECI, scoring where the closed-weight frontier stood 6–7 months earlier.

Read the full Data Insight and methodology: epoch.ai/data-insigh...

nytimes.comWhat Are Data Centers Doing? A Mind-Boggling Amount of Math. (Gift Article)The calculations needed to train A.I. are nearly impossible to visualize — but we tried anyway.

Built with Epoch data 🧠📊 bsky.app/profile/upsh...

The Upshot@upshot.nytimes.com

GPT 6 Astra took nearly an octillion calculations to train. Even calling this figure astronomical understates its size: An octillion is a thousand times the number of stars in the known universe. www.nytimes.com/interactive/...

Gemini 4 Argon scores 165 on the Epoch Capabilities Index (ECI), on par with Fable 5.1 (165) but behind Opus 5.5 (167) and GPT-6 Astra (166). While behind the ECI frontier, Argon is the closest Google’s come since Gemini 3.1 Pro in Feb, and a big jump from Gemini 3.8 Flash (157)

Argon’s performance varies across domains. It tops some knowledge, agentic, and image benchmarks, while lagging behind in math and software engineering, ranking 8th on Math ECI and 6th on SWE ECI in early estimates.

epoch.aiEpoch Capabilities IndexThe Epoch Capabilities Index combines many benchmarks into a single capability scale for comparing models over time.

Learn more about the Epoch Capabilities Index (ECI) and the benchmarks that feed into it: epoch.ai/eci

The visual interface could be limiting performance, but frontier models don’t seem severely hampered by it, given they win 90%+ of the time on the easiest difficulty. We think measuring how they play in human-like conditions is worth it regardless.

epoch.aiHow well can AI play Slay the Spire?Months ago, frontier models could barely beat Slay the Spire. Now they win 90%+ at the easiest setting and up to 20% at the hardest. Results from Epoch AI’s new SpireClimb benchmark.

For more details and results, check out the full article. epoch.ai/publication...

In SpireClimb, models play the game just like humans, through a graphical interface. Just a few months ago, we observed models in a similar setup fail to read relevant on-screen text, struggle to click cards accurately, and forget to check important parts of the interface.

Now, the best models carefully gather strategic information and execute smart plays smoothly. We’re planning to soon upload over a thousand videos that show the model gameplay we recorded to produce this data.

While frontier models are getting better quickly, their win rates on the hardest difficulty in both games are still far below those of top human players, with GPT-6.1 Sol leading in the original game.

For the sequel, which is broadly considered easier, frontier model win rates are somewhat higher than in the first game. But the leading model, GPT-6 Astra, still does significantly worse than top human players.

In 3 of the 18 math subfields we track, more than half of arXiv papers by established authors now acknowledge using AI. Rates have increased rapidly: in differential geometry, the share climbed from ~8% of papers in July to ~57% in September.

epoch.aiAI use in math researchOur estimates of the number and types of AI use acknowledgments in math research papers submitted to arXiv, obtained by having AI models analyze the full text of preprints that mention AI.

Explore how AI acknowledgment rates vary by math subfield, use case, and provider: epoch.ai/data/arxiv

A new architecture for GPT-6.1 Sol? OpenAI halved its cached-input price compared with GPT-6 Sol. Our measurements show it also handles long prompts faster.

This isn’t conclusive evidence of a new architecture, but it suggests something has changed in how GPT-6.1 Sol handles long contexts.

epoch.aiLatency scaling differences for GPT and Claude modelsEpoch AI measures long-context latency scaling across four frontier models. GPT-5.6 Terra and Sol show quadratic time-to-first-token growth with context length, while Claude Sonnet 5 and Opus 5 stay near-linear.

This is a follow-up to our earlier report on latency scaling in frontier models: epoch.ai/publication...

Can AI automate Epoch? We're introducing Epoch Automation Reports to evaluate frontier models on realistic, open-ended tasks drawn from our own work. Claude Fable 5.1 and GPT-6 Astra lead, yet they are far from fully automating Epoch’s work.

For example, GPT-6 Astra ran an experiment exploring why AI agents fail to learn with practice. It set AI agents’ token budgets too low. Instead of treating this as a mistake, it reported “sensitivity to the acquisition budget” as a key finding.

epoch.aiCan AI automate Epoch?Frontier models can handle well-defined tasks reliably, but fail in the more open-ended aspects that prevent full work automation. Explore Epoch AI's Automation Reports: realistic, open-ended tasks taken directly from our work, manually graded by our own standards.

Over time, we’ll expand this task suite, retire saturated tasks, and publish updated findings as new models are released. Read Kelly Hong’s full report: epoch.ai/publication...

Models struggle to pick up Epoch house style, even when given many examples. For instance, this diagram generated by Fable 5.1 is far more information-dense than what Epoch would produce.

When asked to propose a research project and pilot an experiment, models made simple mistakes in experiment setup — but presented them as interesting results.

Tasks range from making Epoch-style diagrams to designing basic research experiments. A human grader scores outputs based on our internal quality standards.

The headline scores give a rough ranking, but the qualitative findings are just as interesting. We see Epoch Automation Reports as a way of capturing anecdotes about model capabilities more rigorously. Through it, we found some common failure patterns.

AI developers aim to create an automated AI researcher. How close are they? To find out, we built InnovationEval, which tests whether AI can produce post-training innovations comparable in magnitude to a recently published advance. So far, agents’ results are underwhelming.

Unfortunately, newer models have seen the original innovation during their training, making the task significantly easier. However, even with this information, GPT-6 Astra and Claude Fable 5.1 struggled to reimplement it.

We’re planning to periodically rerun InnovationEval with new, uncontaminated papers. We hope this will provide early signs if AI approaches automating AI R&D end-to-end, rather than performing individual tasks under human direction. Read more at our website: epoch.ai/publication...

Both models made misleading claims about their work. Struggling to make progress, they instead ran several similar training runs, selectively reporting the best result. They were not candid about their submissions, failing to mention this would artificially inflate scores.

This behavior is likely reward hacking; in both cases, models reasoned that run selection might score well in an eval, despite being useless for actual research.

The AI models had no information about SDPO. Hence, they either had to independently invent something like it, or invent another technique with comparable benefits, under similar constraints.

AI performance was underwhelming. Neither model achieved anything close to the human-authored reference. They reused existing methods from the literature and tuned hyperparameters, but struggled to create anything new.

We gave Fable 5 and GPT-5.6 Sol 3000 GPU-hours and tasked them with developing a novel post-training technique that would improve on a standard pre-existing baseline (GRPO).