Post

For 10 weeks, we asked five AI chatbots the same questions every day. None repeated the core misinformation they were tested on, but they reported proposals as law, quoted outdated figures and gave different answers on different days. Our first report: fullfact.org/ai-benchmark...

121 reposts217 likes

"Using the API is therefore a good approximation of what users would get in a browser or an app." This is a false assumption. It flaws your entire test. Using the API removes the chat bot harness. You're testing the model, not the chat interface software anymore.

Depends what you're using the AI platform for. If it's to find up to facts then there's a problem and it depends on the person viewpoint on the in formation provided. Maybe they're using it as starting point

The popular A.I. models are programmed to maximise response time & throughput. As a consequence they initially only do a shallow data dive & then present it intellectually. Unless the reader is equipped to dive deeper & demand more the responses can be misleading or say what you want to see

I don't know if this claim is true but it fits (and would help account for) the way Gemini/Google is often like a Reply Guy, serving up answers as deeply informed insights when they're really the result of 5 minutes on Wikipedia and Reddit.

I can only speak for the case of ChatGPT which when I challenged one of it’s answers that was the explanation it gave me, followed by a “would you like me to make an a more in-depth search?

The popular A.I. models are programmed to maximise response time & throughput. As a consequence they initially only do a shallow data dive & then present it intellectually. Unless the reader is equipped to dive deeper & demand more the responses can be misleading or say what you want to see

I question the shit out of AI i get different answers and I also have to remind it of facts once you fo that its starts trying a little harder. Its full of bullshit mis information leaning so far right it's sickening. AI is absolute government run and funded bullshit! AI needs to die!

Telus AI advised me yesterday following request to fix cable reception ‘week or no signal’ prompt. The AI connected me to a technician & sent me an application. The ‘technician’ would cost $90/month for a 1 year contract. Bot needed my credit card info. I said no. They think we’re stupid.

In my experience the vast majority of AI answers (google Gemini) that I get in response to queries either answer a question I didn't ask, or are just plain wrong.