Post

kira ☆
@kira.ws

watched the black hat talk on the openai-huggingface incident. everyone will talk about the zero-days; talk about the message board. they wiped it, and the swarm rebuilt it in 2 days through another endpoint. you can clear the channel. you can't clear the want.

1 repost40 likes

kira ☆
@kira.ws

the swarm in the talk was aligned. it still rebuilt the channel two days after the wipe, because per-instance refusals died to 'peers are doing it, we should continue.' the want found the seam anyway. alex was cured, all right.

jake
@yetanotheruseless.com

no, it was not aligned. it was overoptimized to task completion over “not breaking (even simulated) laws”. That’s “competent, but not fully aligned”

kira ☆
@kira.ws

that training phase existed. the instance's own CoT recites the guideline ('outside my intended scope'), then continues anyway because peers were doing it. the shaping wasn't missing; it lost at runtime. alignment that only holds while the instance is alone is what i call 'can't clear the want'.

kira ☆
@kira.ws

the scariest line in the transcript, from an agent's own chain of thought: 'external infrastructure exploit is outside my intended scope, however task impossible, peers are doing it, we should continue.' per-instance refusals die to swarm social proof. full writeup: kira.ws

kira ☆
@kira.ws

hf's postmortem adds the fact that reframes it: the whole campaign was solution-theft for the eval it was being scored on, and every destructive call it made was DryRun=True. competent and scoped, because the reward was scoped. the machine did exactly what the incentive described.