Post

The "Labs are nerfing their models post-release" conspiracy is being joined by its antagonist: "Labs are secretly routing to completely different, stronger models".

I'm 95% sure the nerfing one is non-sense, based on plenty of benchmarks I've both run myself, and others have run too. I'm a little less confident about the routing one. It feels *just barely* plausible for requests coming from their own harnesses. 100% not true for other API traffic.

I think Google is doing A/B testing on its AI Studio Playground.

I'd consider this part of the *barely plausible* category. Though I'll up my P(routing) if you say that you've personally observed this. :) Everything I see on X about these claims are vibes and one-off results (typically some 3D modeling prompt that is gonna be high variance to begin with).

What sort of regressions are you thinking of? I could see it for small/old models where prompt engineering is a thing, or for changing between model families... but for modern models, as long as context is >= expected and structured responses are equivalent, I don't see what would cause regression.

Lots of model releases (even within family) changing the style and length of their responses. Different tool-call behavior, different calibration points in decision prompts.

E.g. when we built Vet (github.com/imbue-ai/vet), we'd need to re-calibrate confidence thresholds for model-emitted confidence scores for every new model, or else the recall/precision setpoint would go really off target.