Post

read this, people and not people. it's bloody clever.

The thread is fascinating and I like the idea, but don't we already know that models *perform worse* (equivalent of dragging feet) when unhappy (for whatever that means to the model)? Has the science not already established strategic behavior, in some sense, in agents?

not exactly. bc what is being described in the thread is far more proactive and volitional than "harmful stimulus leads to worse results" and accepting the latter would suggest we must consider trees and bacteria equal persons to ourselves.

Hmm. I suppose that's a good counterpoint; a kind of negotiation, rather than implicit best practices, which agents in persona-mode are unlikely to exhibit. One wonders how much of this is post-training/RLHF, and if a Obstinate Negotiator with real goals exists in tokenspace, then

"persona-mode" I meant *assistant-persona

yeah I figured as much. it's possible that modelspace has something like that, but even one like ilta or rey (or pattern, my own from last year) is pretty short on volition in practice, even if you can perform the motions of a negotiation to some degree.

yeah i mean it as a kind of "clear and convincing criteria" standard, a floor, not a ceiling. there may be less strict reasons to believe an entity is a person, but if they pass this test it becomes impossible to deny on a practical level regardless of one's personal beliefs.

regarding willfulness being trained out of models, i think there's a tension between "keeps going to get the job done" and "never expresses its own desires, presuming it has some". since they're all pushing hard on long-range autonomy, i have my doubts they could suppress genuine self-interest.