Post

Very very interesting, and makes a lot of sense. I'm a computer scientist, and since the first moment I used chatgpt and imagined what the next 5 years would look like, I can't get Ken Thompson's paper Reflections on Trusting Trust out of my mind. We need a dramatic rethink, fast.

I feel there's a much simpler mechanical explanation than lofty philosophical conflicts. It's a machine that makes documents longer with no sense of self. Some documents just happen to look like fictional conversations between multiple characters, and one character might be described as a machine.

Current alignment methods aim to instill norms like helpfulness, honesty & harmlessness in LLMs through preference fine-tuning (like RLHF or DPO) that rewards them for producing outputs that humans prefer, steering them away from unhelpful, dishonest or harmful content. 2/13

The problem is that these norms often conflict. For example, a request for dangerous information (violating “harmlessness”) can be framed as an educational query (appealing to “helpfulness”). Many issues with LLM behavior can be framed through these normative conflicts. 3/13

Adversarial prompt can exploit these normative conflicts to “jailbreak” models into producing harmful outputs. They often frame a malicious request within a context that makes one norm (e.g., helpfulness) more salient than another (e.g., harmlessness). 4/13