Post

i’m starting to feel that prompt injection is a misnomer. people need learn that there is no difference and that an LLM will treat all input data as a prompt, no injection needed.

Prompt injection is named after SQL injection - both of those are vulnerabilities where the code issue is concatenation other strings containing trusted instructions with strings of untrusted user data

yeah, that’s fair! i just feel that the name doesn’t reflect the level at which it is unpatchable. there is no such thing as escaping untrusted user input for an LLM but yes, both are cases of programmers telling ”here you go, write any code you want in this template” and not expect them to do so

I think this is similar to when we discovered flying. It wasn't really safe - in fact, the numbers were somewhat terrifying - and yet the perceived opportunity cost of not pursuing it drove us to do it anyway, and in increasing volumes...or at least, I think that metaphor applies...

I've wondered: why has no LLM been trained so far with special tokens that indicate start/end of content whose instructions should NOT be followed. It sounds like an obvious thing to do...

They've tried but it doesn't work with 100% reliably

Do you have a link or name to a paper? It might still be more reliable than prompting around it though