Post

At the risk of starting the flame war to end all flame wars... Modern LLMs (GPT-5.1, Claude 4.5, Gemini 3) produce excellent code and can be a significant productivity boost to software engineers who take the time to learn how to effectively apply them - especially if used with coding agent tools

IMO the problem is that we can't coherently measure productivity in software engineering (particularly in and across teams) as a baseline to make sense of this claim.

That part we can and have measured. People consistently overestimate the productivity benefits of using LLMs.

Have you seen any reports demonstrating that aside from the METR one?

The Stanford one, ( youtu.be/tbDDYKRFjhk?...) and both the 2024 and 2025 DORA reports.

The problem is that I have seen enough LLM-assisted work that saved a day for one person, only to add a day of extra work for two other people. So your gut feeling that you saved yourself time can be misleading.

that happens with human code all the time too we call it technical debt, and tell ops people -- i.e. me -- to suck it up when it makes them extra work so i'm not sure we can lay this at the feet of the llms, unless we can prove an overall increase (and gl w/ tech debt metrics: not v measurable)

This is, why IMO, it's more about the size and criticality of the task you use it for and when you choose not to use it. And my criticism is that for most people, no task is off limits.