Post

exactly 💯 on LLMs: they simulate the output we call language, not the cognitive capacity we call language!

I am not sure that I or anyone else confidently understands the difference between those.

Let me try. The outcome can look very similar. While it’s an oversimplification to describe LLMs as next word predictors, for the sake of argument: what’s the difference between the choice made by an LLM and a human? 1/3

An LLM produces the next word based on analysis of what the next word is statistically likely to be given all the vast data it has about previous language use in highly similar contexts, plus guided training by humans. A person chooses the next word based on reasoning & what they want to say. 2/3

It’s a very powerful technique, obviously. It reflects the power of language to store & organise information and relationships. But LLMs are fundamentally based on analysis of previous language use, not the cognitive processes that incite that language. They end up mimicking the latter only./end

Obviously you’re right about the first part of each of these observations, that LLMs are trained on previous human language use. The problem is in contrasting them to human processes such as “reasoning” and “wanting” which actually aren’t well defined. 1

I think it is fine to say that an LLM wants primarily to produce answers that satisfy its users. Humans have a vastly wider range of desires. But saying there is a categorical difference between desires produced by neural connections and hormones vs by models on chips sounds like metaphysics. 2

As for reason, I’m not sure how our neurons achieve that capacity; many humans are bad at it. But it is probably built out of clusters that generalize efforts to predict different things. This is also how AIs develop. I am not sure we can say they don’t reason. 3

If you're interested in an analysis of those statements bsky.app/profile/oliv...

Thanks, you may be right that there is an element of mysterianism here but my question was slightly different: what is the formal description of what reasoning is that excludes what an AI does? Plenty of AIs are trained to do work. Does a neural-network chess AI not understand how to play chess?

OTOH I can read this paper but am not well-educated enough in these fields to understand some of your papers, so maybe I shouldn't be asking questions like this.

Obviously true, but aren’t all “fruitful” (in the “bruto” sense, if we ignore costs, which we shouldn’t) applications to (more)formal subdomains of language, like code, maths, where natural language only serves to convert the task (informally) and the output is code (formal).

Like I 100% agree LLMs simulate language, and language is not formal, but non of that precludes them from having utility? Simulation is very useful when applied appropriately right?

This seems like a fairly unhealthy multi part thread people (maybe you) aren’t enjoying, if so, forget I said anything, I just saw this in my timeline and was curious.

My dad still always manages to play chess with the same rules he uses for language, or anything else for that matter. Those rules are: 1. Whatever means he is in some way winning. Then again he is also the only person I know who has turned out to be completely pointless to try communicating with.

at least I think I can blame him for my (I'm told autism related special interest type) interest in communication systems. I think language is to predict someone else's abstracted cultural framework of abstract ideas, and try to map one's thoughts as defined by one's own such framework onto theirs.

If nobody could understand it, words in a language become meaningless, because it represents 0 possible communication protocols between a sender and a receiver exchanging representations of intended and interpreted messages. not that I could be considered qualified to have opinions on the matter