Post

Everyone is just taking that tweet at face value? Assuming that result is correct, we don't currently have verification it was done just recently - vs. a project they secretly devoted months to then chose to reveal at the most viral moment. Meanwhile, plenty of outside proof for both positions.

computerworld.comAI chatbots are wrong about news 45% of the time, study findsGoogle's Gemini was the worst performer; it had problems with 76% of its responses, according to an international study of public service broadcasters.
Eerke Boiten@eerkeboiten.bsky.social

And the top rating for AI cheating goes to: "In the most extreme case, our model achieved the top rank on a standard chest X-ray question-answering benchmark without access to any images." futurism.com/artificial-i...

sorry but "they actually had this outcome a while ago and sat on it" is a really silly conspiracy theory. I cannot think of an upside and your links are totally unrelated? It's not a surprise that LLMs are bad at news, especially without context? imo very much a misuse of the technology

0 reposts13 likes

I don't think this is what happened here, but I could imagine a company throwing a lot of compute at math problems, sifting out the good results, and then drip feeding them out to keep enthusiasm up and balance bad PR. Seems like a pretty sensible strategy to me, actually.

Good PR ("look how smart our models are!"), distraction from bad PR ("look, our models are doing something cool and not at all sinister"), and generally keeping enthusiasm high ("imagine how many cool math things we could do with 100x more parameters!").