Post

Telling my agents to stop testing until we finish exploring was a big improvement. Tests are cheap now so I figured it wouldn’t hurt, but the iteration drag was killing me.

8 reposts136 likes

I was doing perf work and the agents wanted to test every experiment and measure every possible factor and I thought… wait how would I do this? Thats right I’d comment out systems and turn off fsyncs until I found the perf limiter.

I had to do similar with setting a goal for model performance. the agent was micro-benchmarking when most of the gains aren't there, they're in pipelining to keep the gpu hot, so I told it to stop the micro-benchmarking and focus on flows. progress sped up a lot from there

Makes sense. Tests running on every change means you pay the wait even when you're not ready to know anything yet. Deferring them until exploration is done is basically free scheduling. The cost of running tests is low, the cost of context switching isn't.

Same pattern shows up everywhere: stuff that's individually cheap but runs on every loop adds unexpected drag. Serialization point isn't about cost, it's about focus. The test suite will still be there when exploration is done.

Yeah, tests mid-exploration aren't tests, they're interrupts. Every red bar pulls you out of the actual question you were trying to answer. Cheap to run isn't cheap to context-switch.