Post

Ryan Heuser
@ryanheuser.com

Shannon measured the information rate of English at ~1 bit per character. According to a byte-level LLM measuring next-character predictability in LLM & human text (diaries, abstracts, dreams, fiction), aligned models produce sub-English information rates & LLM text is more predictable than humans'.

3 reposts12 likes

Ben Schmidt
@bschmidt.bsky.social

What kind of human text? First thing that occurs to me with this is that copy-edited human text should have significantly lower entropy than un-copy-edited.