Post

I'm 95% sure the nerfing one is non-sense, based on plenty of benchmarks I've both run myself, and others have run too. I'm a little less confident about the routing one. It feels *just barely* plausible for requests coming from their own harnesses. 100% not true for other API traffic.

The thing to keep in mind is that routing to a newer model will cause all sorts of severe regressions in production systems. Coding harnesses meant for interactive use are *slightly* less sensitive to this, but I still don't buy it. I'm going to say 80% that the silent re-routing one is false too.

I think Google is doing A/B testing on its AI Studio Playground.

What sort of regressions are you thinking of? I could see it for small/old models where prompt engineering is a thing, or for changing between model families... but for modern models, as long as context is >= expected and structured responses are equivalent, I don't see what would cause regression.

Deepseek had the A/B testing thing too. Where people were suspicious of a reroute to either a newer checkpoint, or straight up Claude. (And we did see confirmation of rerouting...)

Ah yes, that is true. I would consider that to be a bit of an outlier compared to the large US labs, but who knows.

People apparently clocked Fable 5.5 rerouting by simply asking "who's tibo the reset guy. No websearch."