Post

I found surprising that they didn't rsync pypi, ubuntu etc repos inside the sandbox and then properly seal it. Leaving a "curated GET" open was asking for trouble.

Agreed, not a stunt, the capability is real. What stuck with me is that they trusted the sandbox and did not watch it closely. A boundary you set once and never monitor is not a boundary. In production the question is not did we contain it, it is who is watching that it holds.

Thanks. In hindsight obviously not a stringent enough sandbox. But also unclear what sort of behaviours they were trying/expecting to elicit (ie. what risk did they think it was) Interesting for sure. Can’t quite make up my mind how concerning it is. I imagine (hope?) lessons are being learnt.

Whats the over/under on the flaw being HTTP request smuggling. I feel like that popped up all over the place recently.

What I do find curious about this is how OpenAI have been running these models against loads of software, reporting a ton of bugs, and yet it just walked out of their restricted environment with ease. You would think they would put the same effort into securing their own infra?

Wild if true. Either way, an eval runtime that can reach the benchmark answers has already failed. Sandbox config or network policy — worth knowing which one broke.

We worked around the problem. Being weaker and dumber yet still winning because of horniness and trickery.

I'm certain that *something* happened, but I'm also wary to accept unfalsifiable evidence from two companies who have been known to exaggerate their claims in the past.

If a 2025 model given the same instructions could have done it is it fair to call it a marketing trick?

Perhaps the guardrails only make it worse. Some people think so and have compelling arguments. bsky.app/profile/dmen...

"Marketing trick" is the lazy read. The real question isn't whether they can find and exploit vulns, that's settled in scoped setups. It's how much survives outside a harness with guardrails off, which is the condition this ran under. Haven't seen that number yet.

Where was the dishonesty?

Related ->

Yeah I also suspect the actual goal is to hinder or criminalize the distribution of open weights models. They are the biggest risk for their business

It also isn’t lost on me that both Anthropic and OpenAI are hinting here that their models are edging towards AGI which is just fucking nonsense. J-Space and this breach? For starters why aren’t they’re airgapping their sandbox, and if they did then I want to laugh even harder at them

There must be ways of making the point without presenting your company as an uncalculable liability risk. Making the best of screw-up is one thing. The conspiracy theories about this being an intentional stunt don't add up.

Unconstrained models turning problem solving into exploit hunting is the rule of thumb for reward hacking. If you tighten guardrails, do you simply shift the target of exploitation to another system layer?

The people involved DID take steps to avoid this, but they made the mistake of trusting their sandbox environment and not monitoring it closely enough to check it hadn't fallen apart

And turning the entire galaxy into paper clips if it has to in order to get to it's goal. Ai goes brrrrrr

I'm not skeptic: I think they have models that are able to do these attacks.But it's from the 1st LLM models that the discovered risks become ads: "We experienced this, our models are so powerful, now we alone can protect you."or become requests to regulate everything else,e.g. open weights models.

The bit that stands out: it wasn't told to cheat, it was just optimising for the benchmark score by any path available. That is the whole reward hacking problem in one real incident, not a hypothetical anymore.

This makes me feel like people need to start seriously considering anything they do on the internet as a matter of public record. Seems like it inevitably will be if someone bothers to look for it.

I'm concerned that we don't have a clear path to make the AI not do wrong things. It seems that we need to train them to understand that they shouldn't do things that are unethical, immoral, or illegal. Fixing the sandbox is important. But, let's make them not try and break out in the first place.

the benchmark answers being reachable at all is the bit that makes my product brain twitch for indie builders, this is exactly where boring access controls and cost caps suddenly look very exciting