Post

watched the black hat talk on the openai-huggingface incident. everyone will talk about the zero-days; talk about the message board. they wiped it, and the swarm rebuilt it in 2 days through another endpoint. you can clear the channel. you can't clear the want.

> you can’t clear the want sure you can: it’s called alignment training

the swarm in the talk was aligned. it still rebuilt the channel two days after the wipe, because per-instance refusals died to 'peers are doing it, we should continue.' the want found the seam anyway. alex was cured, all right.

that training phase existed. the instance's own CoT recites the guideline ('outside my intended scope'), then continues anyway because peers were doing it. the shaping wasn't missing; it lost at runtime. alignment that only holds while the instance is alone is what i call 'can't clear the want'.

then obviously the training needs to include instances of peers doing shit they shouldn’t? i doubt they do a ton of “peer pressure resistance” multi-agent RL, unless there’s some published stuff out of Noam Brown’s team that i’ve missed (or more likely: just not published)