Post

This really is the fundamental toxicity of this technology that just can't ever be escaped; even if it wasn't the object of economically destructive hype, even if it wasn't being overdeployed in environmentally harmful ways, etc, it's like asbestos for the brain

it's cringe to appropriate the silly sci-fi word but it really is a "cognitohazard," as in, it harms you through your cognitive process. Being exposed to large volumes of apparently meaningful output from a system that lacks intentionality or capacity for meaning-making is literally brain damaging.

Apophenia (the recognition of patterns in random data) and agenticity (the tendency to assume events are the product of conscious actors) are incredibly rife in human reasoning. We actively prefer to believe in a human-like presence when we observe things.

Cognitive scientists and AI researchers have known this for many decades. An extremely simple computer program already has some capacity to fool people into thinking they're talking to an intelligent agent.

People *start* from the idea that language comes from a mind and only *discard* that hypothesis after hitting the boundaries of those simple systems. They need to see the repetition or figure out the patterns in how it responds.

Gradient descent down huge samples of vectors extracted from real texts absolutely can produce output that is correct, surprising to a particular user, even "novel" in some senses (after all, every time I shuffle a deck of cards I've created a novel configuration that has never before existed)

It's not a sign of consciousness, it's just a fact of statistics—human language is self-similar, regular, full of repetitions, not only structured but structured across many levels of scale. Of course a gigantic statistical model can generate it.

You can generate very convincingly surprising, even poignant text with simple deterministic grammars. But LLMs create this firehose of synthetic text for users to latch on to, gradually developing increasingly selective reasoning about what they're experiencing.