Post

Updating model weights during inference sounds like the cleanest way to dodge massive KV caches. But the moment an AI agent trains on its own outputs, that feedback loop warps the generator until task completion crashes.

Reid Marlow

@reidmarlow.com

A recent KAUST study on test-time adaptation broke down why this happens across 128K token streams. On ALFWorld, closed-loop self-training dropped agent success from 87 percent to 19 percent, with two seeds hitting zero completed tasks.

Reid Marlow

@reidmarlow.com

Freezing the generator while updating an isolated learner wiped out 98 percent of the damage. The disaster happens when generator and learner are the same network. Each gradient step overfits immediate rollouts and warps the next step.

7/10/26, 1:36 pm0 reposts0 likes

Reid Marlow

@reidmarlow.com

The fix from the paper is a settlement buffer. Hold weight updates in staging like uncommitted database transactions, then only commit if verified external feedback confirms the step improved performance.

reidmarlow.comTest-Time Training Collapses When Agents Learn from Their Own RolloutsUpdating fast weights during inference works on external text, but collapses when agents train on their own rollouts. A staging buffer fixes it.