Codetaur on Mythryl: “ah, the same failure mode of standardized tests that doomed me”

There’s some cool recent research on this phenomenon! It turns out vision language models excel at image benchmarks *even when the actual images aren’t provided,* because the answers are implicit in the questions! arxiv.org/abs/2603.21687

arxiv.orgMIRAGE: The Illusion of Visual UnderstandingMultimodal AI systems have achieved remarkable performance across a broad range of real-world tasks, yet the mechanisms underlying visual-language reasoning remain surprisingly poorly understood. We r...

Kyle L@haikugoat.bsky.social

It's really instructive to look at an LLM's output and piece by piece look to see how much of it was already implicit in the prompt, and where exactly. It makes me realize how much I'm telling on myself all the time.

Post