Post

I've been working with Prime Radiant building a new tool for running small eval suites against models, harnesses, and prompts - it's called "smevals", you can try it with "uvx smevals docs", and I wrote about it here: primeradiant.com/blog/2026/sm...

More on my own blog as well: simonwillison.net/2026/Jul/31/...

Here's an example of the kind of report it can produce having run and graded an evaluation suite against different models static.simonwillison.net/static/2026/...

the failure mode I keep hitting with eval suites is the grader itself drifting, especially when a VLM or LLM is scoring outputs. saw a piece today arguing computer-use agent judges are systematically off. does smevals let you audit or pin the grader separately from the model under test?

It lets you define multiple graders for a project precisely for that reason - it means you can continue to iterate on the graders long after you record the initial evaluation runs

the ability to re-grade recorded runs after the fact is the part I care about most. that decouples grader iteration from expensive re-runs of the agent. can you diff two grader versions on the same run set to see where they disagree?