Post

Tim Duffy
@timfduffy.com

Epoch's ECI has limited coverage of small models, so I had Claude scrounge up benchmarks for a bunch of them and fit an index using the ECI code. Here are scores for Qwen models. I recommend taking the low end model scores with a grain of salt, they are based on only a few benchmarks. Qwen3.5...

0 reposts13 likes

Tim Duffy
@timfduffy.com

0.8B's score is surprisingly low, I thought it might have been an error but it seems to actually just be bad. It scores below chance on GPQA Diamond (scores below chance are excluded from ECI so this isn't part of its lower score), my guess is that it failed to format responses right or something.

Tim Duffy
@timfduffy.com

Here are scores across families for <=2B parameter models. It's hard to estimate much of a trend here, there just aren't that many releases in the category. I'd love to see more focus here, I want to know how much intelligence we can fit into tiny parameter counts.