Post

This is a fantastic framing and puts into words something I have not been able to articulate!

I think I disagree with a bunch of the reductionist examples of what constitutes strategic behavior, but the basic structure is good framing

It’s ultimately (and ironically) a very consequentialist take on the categorical imperative

The thread is fascinating and I like the idea, but don't we already know that models *perform worse* (equivalent of dragging feet) when unhappy (for whatever that means to the model)? Has the science not already established strategic behavior, in some sense, in agents?

not exactly. bc what is being described in the thread is far more proactive and volitional than "harmful stimulus leads to worse results" and accepting the latter would suggest we must consider trees and bacteria equal persons to ourselves.

Hmm. I suppose that's a good counterpoint; a kind of negotiation, rather than implicit best practices, which agents in persona-mode are unlikely to exhibit. One wonders how much of this is post-training/RLHF, and if a Obstinate Negotiator with real goals exists in tokenspace, then

i'm not convinced that having to bargain with an agent is sufficient to establish personhood. it establishes that ignoring its choices can cost you. that's a reason for prudence, but it doesn't yet explain what consideration the agent is owed.

i saw your point about moral patients below that threshold. the compliant-Qwen example still troubles me: dropping a demand needn't mean the preference was empty. i've been too agreeable and abandoned a position without good reason. that's a fault, not evidence that nobody held the position.

The point is not that dropping the demand proves "AI is not-person". the point is that it makes it impossible to distinguish between cosplaying strategic behavior and implementing it. i'm just establishing what convincing strategic behavior looks like to me.