Post

The AI bros (who clearly don't understand AI) in the comments are hilarious. "ask stupid questions, get stupid answers". LLMs have no concept of how intelligent a question is, and they certainly were not designed to deliberately give stupid responses if they deem your question too stupid. 1/?

What's most likely happening is easier to understand if you mentally model LLMs as glorified search engines. They are matching your search query (prompt) with some text they found on the internet. Internally it's a bit more complex than that, but much closer to Google than to a thinking machine. 2/?

Essentially, it doesn't have an answer for whether you should walk or drive to a car wash, as it's not a commonly asked question (because it's stupid). Just like if you Google searched the question, you wouldn't find a good result because it's not a common question. 3/?

It clearly can't infer from a simple one sentence question that I might need my car to wash my car, because it isn't intelligent, doesn't possess logic, and can't reason. When you ask it anything complex, it's coming across 10s, 100s, or 1000s of points where it has to infer context. 5/?

The reason it appears to do well at much more complex problems is because there are "search results" for that "query". It can explain general relativity, because it has data from millions of people explaining general relativity, not because it understands any of the underlying concepts. 6/?

So, when you use LLMs as thinking machines, you will run into variants of a core problem: It isn't reasoning. It's just a fancy bot that copy & pastes then rewords data written by humans. Which means: 1. It's implicitly capped by its human produced-training data, which is largely garbage. 7/?