Post

this isn't cheating, and it's not "absolutely bizarre"! this is just how LLMs work. they are improv machines. they do not have any sense of truth or fiction bc they don't "know" anything. they can't even "know" if they "looked" at an "image." this is totally unsurprising

Eerke Boiten@eerkeboiten.bsky.social

And the top rating for AI cheating goes to: "In the most extreme case, our model achieved the top rank on a standard chest X-ray question-answering benchmark without access to any images." futurism.com/artificial-i...

futurism.comFrontier AI Models Are Doing Something Absolutely Bizarre When Asked to Diagnose Medical X-RaysFrontier AI models generated "detailed image descriptions and elaborate reasoning traces" without ever seeing the images.

Eerke Boiten

@eerkeboiten.bsky.social

30 Apr

The LLM "cheats" on the benchmark; giving the right answers despite having not the questions in full shows that it cheats by reproducing an answer rather than doing any "analysis". This may be obvious to you, but it is not obvious from the claims made about "potential" let alone "intelligence".

Amy Hoy: Pro Edition

@amy.stackingthebricks.com

it can’t “cheat” because it is not taking a test or giving answers. it is simply playing improv.

2/5/26, 5:18 am0 reposts1 like

Amy Hoy: Pro Edition

@amy.stackingthebricks.com

2 May

also i have no clue what you mean with that “it may be obvious to you” jab. you think *researchers* shouldn’t know this? because wow that’s really something

Eerke Boiten

@eerkeboiten.bsky.social

2 May

I assumed it was obvious to you. "While it may be obvious to you, it is not obvious to everyone" is not a jab at you.