Post

@larryosterman.github.io

Can someone help me understand how an LLM can hack into a web site? Fundamentally an LLM is a chat bot. It takes text in and spits text out. The text out can be an instruction to a host to create a file or fetch a url, or even to write a program.

Larry Osterman

@larryosterman.github.io

But ultimately everything done by the LLM is an action performed by the host. And the host can and should have guardrails. And the host can be in a vm, with egress filters, firewall rules, etc. Is the claim that the llm first broke out of the sandbox it was in? If so, that seems like a big deal.

Larry Osterman

@larryosterman.github.io

I'm also confused about the LLM ignoring the denial of a request. Ultimately the host either performs the actions requested by the LLM or it fails the request to perform the action. The host doesn't ever say 'no, I don't want you to do that' - it just fails the operation.

28/9/26, 4:36 pm0 reposts2 likes

Larry Osterman

@larryosterman.github.io

I could see the LLM trying something different after the first action failed. But that's not "ignoring a denied request", that's "working around a failure"

🌍 Duncan Kimpton 🛠️

@dgkimpton.bsky.social

It's a loop basically, the "agent" program takes a prompt like "solve this problem" and asks the llm how to do it, then does what it asks, gives it the output, asks what next, does it, around and around until the problem is solved. The only human interaction was the first step defining the problem.

Steve Syfuhs

@syfuhs.net

There are two fundamental problems. The first is that most guardrails are instructions to the LLM "don't do X". That's because problem two is that the sandbox is pretty dumb about what is safe and isn't safe which is how you end up with /yolo or /allow-all otherwise it's gated on a human ACKing.

Larry Osterman

@larryosterman.github.io

Doesn't that require you running hacking experiments without any sandboxes or at least monitoring? That's the part that confuses me. How do you run experiments without any monitoring?

Eric Lippert

@ericlippert.com

You're granting too much good will towards the criminals who develop and run these tools. They built a tool to illegally defeat security systems and successfully ran it. That the tool has a scripting system that generates its own scripts at random is a gimmick for marketing purposes.

Steve Syfuhs

@syfuhs.net

The problem is monitoring for what? You've seen the transcripts of a normal operation. It's ridiculously long and self-reflecting and trying to extract dangerous or sketchy actions from that is needle in haystack problems. Then, throw a hundred of these things in and try to see what each one does.

Steve Syfuhs

@syfuhs.net

The sandboxes are getting better but the LLM is trained to get the problem solved and "don't do X" is more of a suggestion so it's this ratcheting problem. What might look like a perfectly safe operation to the sandbox is just one tool in the solution to the problem.

Steve Syfuhs

@syfuhs.net

And to complicate matters, most sandboxes are fundamentally local to the machine, not the universe in which it can plausibly do stuff. Local systems are a PITA to protect and distributed systems are impossible.