Post

I wrote about the most ambitious form of AI-assisted software development I've seen yet - Strong DM's "Software Factory" approach, where two of the guiding principles are "Code must not be written by humans" and "Code must not be reviewed by humans" simonwillison.net/2026/Feb/7/s...

simonwillison.netHow StrongDM’s AI team build serious software without even looking at the codeLast week I hinted at a demo I had seen from a team implementing what Dan Shapiro called the Dark Factory level of AI adoption, where no human even looks …

22 reposts150 likes

> I’m particularly invested in the question of what it takes to have agents prove that their code works without needing to review every line of code they produce. ever looked at formal verification based methods? @dollspace.gay on here has tried it to good results

No I've not dug into those at all yet - they do feel a bit more accessible now than they used to, but I'll still need to spend a bunch of time to get up to speed on them

it's the Truman Show approach to testing. i love when engineers go-to-town with LLMs it blows your mind! (yeah tradeoff is it blows the wallet too..)

Automating QA is a thing on my to-do list. I've got quite far in the thinking and, for pure speed practicality, I'm looking at systems that minimise agent use. IE, they run many times and turn to the agent when something unexpected happens.

I actually ran higher than $1000 in my 1 day toy test. The biggest cost was not the initial code, but the review loop which took multiple iterations. I was closer to $2500 for this. If I ignored for and finish I could have shaved off a couple of hundred I'm sure.

Musings, Sometimes@musings-sometimes.bsky.social

I actually gave this a spin on a reasonably complex Go task. Very impressive. I still had to push Claude through the review process about 10 times. But something that would have taken a month took a day.

Without divulging the details since I really want to stay anon here, the task was implementing an AST for a subset of Lua with a lot of wrapped infra code for AWS+ UI. I didn't want to offload all the parsing to the LLM because it's a tractable problem.

I think a lot of these claims are a pissing contest. Reminds me of people bragging about their sex life in high school despite not having one.

Because at that point you’re not *necessarily* going faster than other cheaper human-involved software engineering practices. The 1000/day setup might live in speed/quality/cost Pareto front with strongdm, but almost certainly doesn’t at the equivalent single max plan cost

1000/day is 365k/year, or about 1 fte at ~180k/yr salary with the simplest adjustment for company costs to hire someone. If you convert token costs to subscriptions, there’s around a 10x subsidy. So you’re comparing 1000/day in token spend to 30 max plans. Your 1max plan idea is 1/30 of a 180k dev

The $1k/day number reads differently depending on whether you think token costs will keep dropping or plateau. If inference gets 10x cheaper in 18 months, today's burn rate is actually buying future-proofed institutional muscle memory. The bet is on the trajectory, not the snapshot.

I think a very (excessively?) generous reading of that is $1000/engineer/day *at the current cost per token which will eventually come down*

One day I see a bug in a product I use, submit a 3 line response, and the fix is released the next day … instead of never (Apple’s current standard) It will be interesting to review the automated code a few years from now. Will it be all a big mess of goto statements?

Perhaps a “bunch of goto statements” is simply the next viable language pattern — one that LLMs inherited to move beyond human coding style. As long as it works, who will really care? Maybe only a few old-school nerds like us 😉

The folks at Truong DM are great! I’ve been working with them on Cedar (you may find that interesting too, especially given the AI friendly, automated reasoning) and was unaware of this

We got so good at this (which means we are so so far behind people like Anthropic) that we were 3-4 weeks behind on PRs. Reminds me of MBA use cases about IBM when Agile came out and QA couldn’t keep up with the new speed

Interesting; it's a bit unclear to me how the scenario testing is setup, any links/resources to learn more?