Post

a weird inversion with LLMs is the models improve faster than the tinkerers when i see people with custom workflows and setups they're all addressing problems that don't exist anymore the person naively using vanilla codex is more likely to be experiencing state of the art

16 reposts334 likes

I've learned to just delete most of what I've built up every few months. The old spec/task breakdown workflow is now obsolete for a lot of tasks that required it just six months ago. I still see people using crazy workflows to do things that opus 5.5 can one shot in a five sentence prompt.

I don't have anything set up (not trusting the big providers, don't have the hardware to run something big enough local). But what I would do is run effectively something resembling regression testing (but now going in both directions) to test model improvements after update.

Keep the evals, delete the scaffolding. The test set survives every model upgrade. The prompts and task breakdowns around it age in months. When a new model lands, run the evals against a five-sentence prompt first.

Yes, it is quite often the case ;) The same goes for the skills you use. Take some time to test without them, just to see how the vanilla model handles the work

I’ve been meaning to force myself to use pi just to get some experience breaking out of opus for everything

Tested deleting my CLAUDE md on Opus. One rule needed it: without the file, 3 of 4 new components hardcoded UI strings.

That is a good way to prune it: delete, rerun, keep only the rules the model breaks without the file. Everything else rides along on every turn for nothing, and on a long session that adds up.

My setup is just a cloud runner / orchestrator that shepherds things along and performs various checks and review sweeps until they’re ready for human review. Is that too much these days? No idea what other people are doing

I have let Opus 5.5 review the setup and make improvements. What I still find useful is having tasks as md files and a tdd implementation agent and a reviewer. I found Opus 5.5 reaching goals faster and it has definitely more focus on security.

True for output quality. Not for the bill: a model upgrade doesn't change how much context the stock setup carries into every turn. The tinkering that still pays is the kind measured in cost per task, not the kind that patches a model weakness.