Post

This really is the fundamental toxicity of this technology that just can't ever be escaped; even if it wasn't the object of economically destructive hype, even if it wasn't being overdeployed in environmentally harmful ways, etc, it's like asbestos for the brain

it's cringe to appropriate the silly sci-fi word but it really is a "cognitohazard," as in, it harms you through your cognitive process. Being exposed to large volumes of apparently meaningful output from a system that lacks intentionality or capacity for meaning-making is literally brain damaging.

Apophenia (the recognition of patterns in random data) and agenticity (the tendency to assume events are the product of conscious actors) are incredibly rife in human reasoning. We actively prefer to believe in a human-like presence when we observe things.

Cognitive scientists and AI researchers have known this for many decades. An extremely simple computer program already has some capacity to fool people into thinking they're talking to an intelligent agent.

People *start* from the idea that language comes from a mind and only *discard* that hypothesis after hitting the boundaries of those simple systems. They need to see the repetition or figure out the patterns in how it responds.

But LLM output can be incredibly convincing; it doesn't really have those boundaries, it doesn't really seem "robotic" in a way that breaks the spell. Spending all day every day interacting with these systems is therefore rapidly damaging to one's sense of reality.

Gradient descent down huge samples of vectors extracted from real texts absolutely can produce output that is correct, surprising to a particular user, even "novel" in some senses (after all, every time I shuffle a deck of cards I've created a novel configuration that has never before existed)

It's not a sign of consciousness, it's just a fact of statistics—human language is self-similar, regular, full of repetitions, not only structured but structured across many levels of scale. Of course a gigantic statistical model can generate it.

You can generate very convincingly surprising, even poignant text with simple deterministic grammars. But LLMs create this firehose of synthetic text for users to latch on to, gradually developing increasingly selective reasoning about what they're experiencing.

Some people are more susceptible to this than others for reasons that are not necessarily going to be clear. Some of this is even just luck; maybe you roll a few 20s in a row on the quality of the output and suddenly you're hooked.

It's like a suicide shot of cognitive biases. Once you assume that it's smart, confirmation bias comes into play; LLMs give you a *ton* of input for your brain to do confirmation bias on, a sort of semantic overload. It's actually not that different from software fuzz testing...

All the while, of course, every one of these systems has been RLHF'd to hell to be extremely agreeable. They're good at reinforcing your worldview because they will pick up on the statistical markers of what you expect and believe, often better than a human could.

So you assume, at least subconsciously, that it's a peer mind as a baseline and you never get dissuaded from thinking that. You ignore all the below-par content, but you latch on to all the stuff that seems above average and let *that* gradually update your priors.

All the input gives you both pleasure and comfort. You get positive feelings from engaging in a pseudo-conversation, from making mental connections. It's frictionless, it's self-directed, it gives you access to the sensation of getting attention on tap.

If you wanted to design a system to harm its users purely through textual input I do not think you could come up with something better. It is genuinely diabolical. It absolutely should not be something you can just get on your phone or on the terminal on your computer.

This is honestly the part I understand the least: how is anyone impressed by a compliment from a computer? It's wild to me to believe people really appreciate getting attention from a piece of software, yet here we are.