Post

Full Fact
@fullfact.org

1/ For 10 weeks, our fact checkers and data scientists asked five leading AI chatbots the same 22 questions every day: Gemini 3.1, ChatGPT, Grok, DeepSeek and Claude. None repeated the core misinformation they were tested on. But we found several points of failure 🧵

Full Fact
@fullfact.org

2/ Firstly, rules not yet in force were reported as law. Chatbots said bank cards could be used as voter ID, and that a £10,000 asylum rule was in force, despite it not having been adopted.

Full Fact
@fullfact.org

3/ Secondly, outdated figures were presented as current. Several models quoted a weekly asylum support rate of £8.86 after it had risen to £9.95.

Full Fact
@fullfact.org

4/ We also saw UK-wide answers that ignored differences between the nations. Voter ID rules vary across the UK, but models often left out the caveat needed to make their answers accurate.

0 reposts2 likes

Full Fact
@fullfact.org

5/ And finally, the same question got different answers on different days. One model correctly said England last won a major football trophy in 2025, but said 2022 on other days.

Full Fact
@fullfact.org

6/ We’re calling on AI companies to make sure models: ▪️check live, authoritative sources for time-sensitive facts ▪️distinguish clearly between rules in force and proposals ▪️keep essential caveats, even if answers get longer ▪️can account for the sources they cite

Aldous Gray
@1015saturdaynight.bsky.social

I asked Gemini for a response to this thread, and it said: "This is a really insightful breakdown from Full Fact. The inconsistency point—giving different answers to the same query on different days—is particularly troublesome given how authoritative AI models sound regardless of accuracy.