Post

The "Labs are nerfing their models post-release" conspiracy is being joined by its antagonist: "Labs are secretly routing to completely different, stronger models".

I'm 95% sure the nerfing one is non-sense, based on plenty of benchmarks I've both run myself, and others have run too. I'm a little less confident about the routing one. It feels *just barely* plausible for requests coming from their own harnesses. 100% not true for other API traffic.

The thing to keep in mind is that routing to a newer model will cause all sorts of severe regressions in production systems. Coding harnesses meant for interactive use are *slightly* less sensitive to this, but I still don't buy it. I'm going to say 80% that the silent re-routing one is false too.

I think Google is doing A/B testing on its AI Studio Playground.