Post

New (very short) paper posted! Researchers have been exploring using LLMs as substitutes for human subjects. We find that LLMs are not reliable substitutes, and generate samples that are statistically impossible almost half the time, about what you'd expect from a random number generator.

Populations have a very simple property: if you split them up, the average over everyone has to sit between the average of the subgroups. LLMs fail this very simple test a huge amount of the time. We think this is a pretty big problem for doing inference with LLM-simulated panels.

What's nice about this is that you can run the test for any setting: you don't need any human data to do it. That's important for evaluating whether synthetic samples work in new situations or experiments, since otherwise you can't evaluate the performance without running the experiment!