Post

At the risk of starting the flame war to end all flame wars... Modern LLMs (GPT-5.1, Claude 4.5, Gemini 3) produce excellent code and can be a significant productivity boost to software engineers who take the time to learn how to effectively apply them - especially if used with coding agent tools

IMO the problem is that we can't coherently measure productivity in software engineering (particularly in and across teams) as a baseline to make sense of this claim.

You're right we can't, but on a subjective level it sometimes gets hard to discount if you have enough moment where you go, "Wow, this just saved me 2 weeks of work". There's a lot of opposite moments as well, but you can get better at avoiding them by getting a feel for how/when to use LLMs.

That part we can and have measured. People consistently overestimate the productivity benefits of using LLMs.

Have you seen any reports demonstrating that aside from the METR one?

The Stanford one, ( youtu.be/tbDDYKRFjhk?...) and both the 2024 and 2025 DORA reports.

Yes. The biggest red flag for me is that everyone is talking about how they are 2-10x more productive but some how that doesn’t translate in to value creation for the company. The economy is not booming, neither is the number of dev jobs.