Post

Can someone help me understand how an LLM can hack into a web site? Fundamentally an LLM is a chat bot. It takes text in and spits text out. The text out can be an instruction to a host to create a file or fetch a url, or even to write a program.

But ultimately everything done by the LLM is an action performed by the host. And the host can and should have guardrails. And the host can be in a vm, with egress filters, firewall rules, etc. Is the claim that the llm first broke out of the sandbox it was in? If so, that seems like a big deal.

I'm also confused about the LLM ignoring the denial of a request. Ultimately the host either performs the actions requested by the LLM or it fails the request to perform the action. The host doesn't ever say 'no, I don't want you to do that' - it just fails the operation.

I could see the LLM trying something different after the first action failed. But that's not "ignoring a denied request", that's "working around a failure"

There are two fundamental problems. The first is that most guardrails are instructions to the LLM "don't do X". That's because problem two is that the sandbox is pretty dumb about what is safe and isn't safe which is how you end up with /yolo or /allow-all otherwise it's gated on a human ACKing.

You're granting too much good will towards the criminals who develop and run these tools. They built a tool to illegally defeat security systems and successfully ran it. That the tool has a scripting system that generates its own scripts at random is a gimmick for marketing purposes.

The problem is monitoring for what? You've seen the transcripts of a normal operation. It's ridiculously long and self-reflecting and trying to extract dangerous or sketchy actions from that is needle in haystack problems. Then, throw a hundred of these things in and try to see what each one does.