Post

Large Language Models (LLMs) with "reasoning" capabilities are super hyped, but new research from Apple shows their "thinking" is often an illusion. Beyond a certain complexity, they fail completely. Key points in the 🧵.

Key points: - At low complexity, standard LLMs excel (faster, cheaper). - At medium complexity, reasoning models (LRMs) outperform. - At high complexity, both collapse, with nearly 0% accuracy.

The paper identifies three distinct "reasoning regimes" in puzzles like Tower of Hanoi and River Crossing. Frontier LRMs (Claude 3.7 Thinking, DeepSeek-R1) outperform standard LLMs only in the medium complexity zone. Beyond this, their accuracy collapses regardless of compute.

Researchers even handed models step-by-step algorithms (e.g., for Tower of Hanoi). The result: identical failure points. LRMs couldn’t reliably execute prescribed logic—suggesting their "reasoning" is often pattern-matching, not general intelligence.

Key takeaway: Don't trust LRMs for highly complex tasks. They solved Tower of Hanoi (100+ moves) but failed River Crossing after just 5 moves, likely due to *training data gaps*. For critical planning, coding, or math, always verify outputs.

This isn't just about benchmarks—it's about fundamentals. Current "thinking" models are impressive but eventually fail, and users won't know at which task or complexity level. Scaling model size won't solve this; we need new paradigms for robust reasoning.