Post

The "Labs are nerfing their models post-release" conspiracy is being joined by its antagonist: "Labs are secretly routing to completely different, stronger models".

I'm 95% sure the nerfing one is non-sense, based on plenty of benchmarks I've both run myself, and others have run too. I'm a little less confident about the routing one. It feels *just barely* plausible for requests coming from their own harnesses. 100% not true for other API traffic.

The thing to keep in mind is that routing to a newer model will cause all sorts of severe regressions in production systems. Coding harnesses meant for interactive use are *slightly* less sensitive to this, but I still don't buy it. I'm going to say 80% that the silent re-routing one is false too.

What sort of regressions are you thinking of? I could see it for small/old models where prompt engineering is a thing, or for changing between model families... but for modern models, as long as context is >= expected and structured responses are equivalent, I don't see what would cause regression.

E.g. when we built Vet (github.com/imbue-ai/vet), we'd need to re-calibrate confidence thresholds for model-emitted confidence scores for every new model, or else the recall/precision setpoint would go really off target.