Post

A few days after Blue Prince came out, I fed one of them some randomly chosen examples of the harder three-box parlour puzzles (blue-prince.fandom.com/wiki/Parlor_...). I got a pretty high rate of getting the right answer. Any thoughts on what was going on there? I don't think it was regurgitation.

The reason it appears to do well at much more complex problems is because there are "search results" for that "query". It can explain general relativity, because it has data from millions of people explaining general relativity, not because it understands any of the underlying concepts. 6/?

The problem is that LLMs don't know what a car is, what a car wash is, what driving is, what walking is, what distance is, and how these things relate. It has no conception of the physical world. Because as you point out, it has no conception, no thought, no consciousness at all.

There's also the problem that if someone does ask a stupid question like that in the training data, it's in the context of a joke, where the answer is deliberately wrong but the LLM cannot distinguish that.

"Since you are getting your car washed, driving is the practical choice so you can actually bring the vehicle to the service area. Additionally, the nearest car wash is about 2.7 miles away, which would be a fairly long walk." Seems like you're wrong

Many people too young to know about early days of Alta Vista, Hotbot, Lycos where the operators in your query were make-or-break for useful results. Prompt construction is very similar in this regard...except 5x the work.