Post

Biggest question: where does orchestration stop and product judgment start? A software factory can automate PRs, tests, deploy triage, etc., but the hard part is deciding what not to build.

Excellent point. That's exactly why I separate the intent layer from the implementation layer. I'm spending most of my time in the intent layer and trying to automate everything else.

How do you test and optimize your factory? Do you use evals? Tracing? Do you delegate tasks to cheaper models?

Building and optimizing the factory is my job now.

What prevents the factory from being able to build and optimize itself?

- what % of code do you still read? - how do you measure the speed up/ROI on tokens? - where do the bottlenecks shift to? - did you invent new ways to do QA?

Great questions. I'd say I read a relatively low percentage of the code, and I'm rather looking for signals that tell me something is wrong, the same way how a chef in a restaurant might taste something a little bit but doesn't taste everything that goes out of the kitchen.

I'm doing pretty extensive bottom-up measurements with an OpenTelemetry stack where I can see pretty clearly how things get faster and more efficient, and I also get a better sense on how to measure complexity.

would be cool to include how much you spend on ai as part of conversation and if your approach would be affected if token costs changed