Post

Just inadvertently discovered that a whole bunch of research on AI responses may be invalid. Apparently, even with memory off, ChatGPT is still giving contextual information, including the name of the user! It used my full name casually in conversation; which means that data is in the model.

To clarify why this is a big deal, the way that AI works is that anything put in the prompt changes the weights and thus the likelihood of certain responses in the output. If there’s no information about someone on the web, these changes should be small, but if so, responses can differ drastically.

This also has huge implications for the potential for racial bias, as users’ names might end up differentially, weighting different responses and textual styles or even information for people whose names are ethically or racially coded.

And it means there’s a potential that an AI conversation without personal identifiers will be incorporated into the learning system in a way that is personalized, both because potential that the AI mentions a real name like it did for me or because of differential weights.