Post

Large Language Models (LLMs) with "reasoning" capabilities are super hyped, but new research from Apple shows their "thinking" is often an illusion. Beyond a certain complexity, they fail completely. Key points in the 🧵.

Key points: - At low complexity, standard LLMs excel (faster, cheaper). - At medium complexity, reasoning models (LRMs) outperform. - At high complexity, both collapse, with nearly 0% accuracy.

Worst of all, LRMs reduce their reasoning effort as problems get harder. Despite having spare token budgets, they underthink when complexity peaks, like a student giving up on an exam’s toughest question—even with extra time left 😅

Researchers even handed models step-by-step algorithms (e.g., for Tower of Hanoi). The result: identical failure points. LRMs couldn’t reliably execute prescribed logic—suggesting their "reasoning" is often pattern-matching, not general intelligence.

Key takeaway: Don't trust LRMs for highly complex tasks. They solved Tower of Hanoi (100+ moves) but failed River Crossing after just 5 moves, likely due to *training data gaps*. For critical planning, coding, or math, always verify outputs.