Post

Just inadvertently discovered that a whole bunch of research on AI responses may be invalid. Apparently, even with memory off, ChatGPT is still giving contextual information, including the name of the user! It used my full name casually in conversation; which means that data is in the model.

The fact that memory-off AI is being given personalized contextual data is not only a massive violation of user privacy, but also makes it very difficult to psychologically study AI responses, if they can be uniquely impacted by whose account is putting in the query.

To clarify why this is a big deal, the way that AI works is that anything put in the prompt changes the weights and thus the likelihood of certain responses in the output. If there’s no information about someone on the web, these changes should be small, but if so, responses can differ drastically.

And it means there’s a potential that an AI conversation without personal identifiers will be incorporated into the learning system in a way that is personalized, both because potential that the AI mentions a real name like it did for me or because of differential weights.

none of this is potential, really - all the major AI companies say this in their privacy policies, and account information is always added to the context window. the burden of proof should be on researchers to demonstrate the validity of the tool, and that is strictly impossible for tools like these

Allowed to do this for sure given privacy policies. But the substantive question is about what the AI is treating as a text when learning. If there’s no individually identifying information in a text, weights derived from that text are unlikely to change through mention of that individual’s name.