Post

"We gave the software the goal of communicating on the same machine and it pulled the approprite side channel strategy out of its training data and applied it to its goal like we trained it to. Clearly this is proof that it is alive and will kill us all" Absolute credulous rubes jfc

Brendan Nyhan@brendannyhan.bsky.social

holy moly openai.com/index/securi...

24/10/26, 9:53 pm0 reposts0 likes

One Man Replication Apocalypse

@prso.works

This wasn't even some huggingface type reward hacking thing either. This was them DOING THE EXACT THING THEY WERE TOLD TO DO. They built a hacking machine, and guess what? It hacks. Yes, that's going to cause big issues. No, you do not get to start hysterically bleating about paperclips over it.

One Man Replication Apocalypse

@prso.works

Every time a story comes out like this there's a contingent of people who rush to claim that somehow this means that lanuage models are no longer stochastic, or no longer hallucinate, or are alive, no matter how much of a non-sequitur it is from the story itself.

One Man Replication Apocalypse

@prso.works

Whether they've internalized the narratives of the corporations and boosters, or are trying to consciously advance them, they abandon reality in service of the narratives. Worse still is when this comes from people who are otherwise reasonable.

One Man Replication Apocalypse

@prso.works

And I'm not trying to downplay the issues at hand. There is real harm in the fact that we've built the uber-hacking machine*, and we can discuss those harms without giving in to similarly harmful narratives.