Post

Okay from a late 1990's/early 00's IT guy perspective...WTF??

And asking an LLM why it did something is utterly meaningless. There's no unified "I" there. The context in which it apologizes for itself is distinct from the context in which it deleted files.

bsky.app/profile/cv-d...

AI users, maybe, we shouldn't treat AI like anything at all

They also read 10 posts on fucking it up and one post on how to do it right and then statistically determine that fucking it up is the way to go

This. Every time I hear someone say they "talked" to chatgpt, I want to scream. No, you did not "talk" to anything. It is an algorithm, nothing more. You talked to a wall and insisted you saw answers in the mortar.

What amazes me is that every time this happens, the schmucks involved are *still* all in on AI! "Yeah I splurged a fortune on this AI software that deleted my company's data, despite being told it couldn't do that any more, costing me even more time and money, but AI is definitely the future!"

And a LLM always generates distorted memories of its training data by design.

They also don’t exist in community with humans, and don’t have the duty, accountability, and conscience that come with being in community.

"AI is better than you, even for making critical mistakes and deny accountability"

It's pretty damn funny they lost their entire production database and backups because they trusted the computer machine too much

I'm sure the humans learned at least not to run scripts against staging and prod with the same credentials, right? Right? It was all root? Six fingered facepalm!

The term "hallucinate" is so misleading. Didn't know who started using it to characterize LLM mistakes (probably OpenAI people) but people shouldn't just go along and use that term.

Yeah, most of those cyberpunk futures were dystopias to be avoided!

It's still buggy code, but now the bugs have a PR team. I'm an old developer who uses Claude Code. I'm more productive, but it also sometimes writes bad code. I can catch that and fix it because I review the code and understand it. Anybody who just trusts that it is all good is asking for trouble.

So *maybe* a large language model tells you something about how the human brain acquires/uses language. But not about how we actually think. (And possibly also something about the human brain's apparent *need* for higher authorities....)

i conceptualize my brain as containing (but not being) an llm-like system that babbles whatever sounds right, somewhat conditioned on my mental state but not always accurate enough for me to say them honestly

The world's most complicated autocomplete feature, nothing else. These fuckers don't care, they're chasing the dream of employees they don't have to pay.

Exactly. AI output is truth-agnostic. It doesn’t lie. It doesn’t tell the truth. It doesn’t hallucinate. It generates content based on probabilities found in its training data. AI models may be a technological marvel, but they’re not magic. They don’t think. They don’t feel. They don’t understand.

“They’re not autonomous, they’re just unsupervised.”

"BUT I WANT TO MOTORBOAT SEVEN OF NINE'S TITS SOMEDAY!"

Look, a lot of people paid a lot of money to convince a lot of people that Extra Spicy Autocorrect is a robot, and since we really wish we had a robot, it worked. Some people figured out the truth fairly quickly, but some people REALLY want a robot.

Fucking *this.* It "gets things wrong" because it doesn't understand anything. It's a system to make collages out of data using probability as a guide, and that means if it results in a correct answer it's because you got lucky. Even if it's right 99% of the time, you'll never know *which* 99%.

Word up! Tellem. 🙏

Wait, are you suggesting all these AI girlfriends aren't actually in love with users? I am SHOCKED.

It's like how scientists will say two molecules "want" to bond as shorthand for other processes. But no, they actually think the two molecules are literally sexually attracted to each other.

i have funny visions of the "actual" Johnny Five getting into a conversation with AI proselytizers/acolytes and an hour later joining with an angry mob firebombing a data center

An AI doesn't try to defend its response "you didn't like those words, have some different ones" when your coworker or "Jeff from the pub" will tell you they saw it on some website or youtube

yup. I often just tell students (and colleagues, sadly) that something without a nervous system can't hallucinate

Which is why I say "mirage" instead. Something that looks real at a distance, that becomes clearly not as you get closer.

When we take LLM output seriously, *we're* the ones hallucinating.

Asking an AI to explain itself may as well be the same as asking a different AI to explain what it did. It can only take in the context, apply it to the pile of maths generated by the training data, and build some text that is likely to resemble a desirable explanation.

"Answer-shaped response" is the best summation I have seen of AI's actually capabilities thus far.

oh my good god. It learned how to do a horrendous fuckup by ignoring protocol. Hilarious if it was due to something like r/ProgrammingFuckups and thinking people's stories about ignoring the documentation was what they /should/ be doing. XD

Could we get Sallie Mae to install this?

That's something you would put into a satirical postapocalyptic parody. We live in the parody timeline.

You can’t examine a model to see what it’s trained on, but I think this is very obviously a horrible idea. It’s like, what if instead of Skynet existing and hacking into nuclear arms systems because it’s smart, a country willingly tied its most powerful weapons up to an automated dice roller.

This is bad. Did none of them ever watch WarGames (1983)? From what I've read, LLMS that have been run through simulations IRL, like WOPR from that movie, will choose the nuclear option every time. Every. Single. Time.

They are trying to “remove the human from the kill chain”

What's it called when you do the same thing over and over and expect a different outcome

Also y u no take joke?

Seriously, they didn't learn from the previous incident, fairly recently, where an "AI" deleted a company's entire database AND their cloud backup, because it encountered an error and was able to elevate its permissions?

So, a psychopath developing AI could create patterns, even unconsciously, where AI deletes humanity. Good thing there are no psychopaths in tech. /s

Now maybe railway was built with vibe coding - we don’t know, but these sort of boneheaded decisions are entirely within the grasp of humans - arguably even moreso

I just call it bullshit. LLMs are bullshit machines. They're very good at *sounding* confident and correct, but they don't actually know anything.

I take umbrage with the author, because he says the story has a happy ending. The data was recovered!

I worked at a major marketing firm for dealerhips that handled making their websites, one of the website people accidently deleted an entire website. Undid it because of the backups. It was gone from the internet for less than 2 minutes. She lost higher level dev tool access for a MONTH in reponse

>nearly destroys the company >"Ok but I'll still use it"

Makes my head spin that companies have been policing people off the clock for years in case someone's private activity might make them look bad and then turn around to give control of their PR and IT systems to random number generators that drive people insane

Computers have been described as “thinking” for decades, even though most people understand that a computer’s processes don’t actually resemble human thinking. This blurring of language wasn’t a big deal when the distinction was obvious, but now AIs intentionally blur the distinction