Post

my wife googled “united airlines customer service” the other day and the AI summary seemed legit enough so she called the number that it listed. she got like 30 seconds into the call before they were asking for her BANK INFO. a scammer had gotten their number prominent enough that the AI grabbed it.

1.4K reposts3.9K likes

I can't help wondering if it's any harder to train the old onebox classifiers to do that. (I've seen some real disasters that I'm not going to repeat here.)

I suppose if they start with the same sequence of digits, it might just take insinuating itself into the training material. It seems unlikely that there are a large quantity of phone numbers that would have that kind of prefix.

Regardless, it's deeply frustrating that likely almost every single thing that someone caught and fixed in the old Google answers is going to have to be caught and fixed *again* because it's a new training set and model.

My in-laws got scammed this way and the scammers got them to call my wife to "verify their bank details" which included making a ton of small transactions back and forth and then a big transaction. All because my mother-in-law wanted to call amazon to return a package.

The scammers had my in-laws install remote admin software on their phones and then used that to keep texting my wife until the in-laws realized and shut their phones off.

While we are at it, a lot of blame goes to the companies actively HIDING their real phone numbers for years to try to make you use shitty chats so searching for them was the only way to get a person

Same thing happened to me googling Apple Customer Service then they got my mom doing the same with Netflix Customer Service and it even had a hotlink to call them from the search. They ALMOST got me and got even closer to ALMOST getting my mom. How is this not regulated yet?

Takeout spots near me have signs warning there are no refunds on fraudulent orders. Fake ordering websites are ranked higher than legit ones and linked on the Google map listings; businesses can’t get them removed. Madness.

Yikes 😬 So glad all the search engines are broken now! I was looking up a an ancient city to see how to spell the name and weirdly ai was the only one who would take me to the wiki link and you can’t tell me that’s not on purpose

That happened quicker then I thought tbh. It was only a matter of time before scammers would try to hijack AI (or as I call it Artificial Stupidity), but never in 7 moons would I've thought that it'd happen today with the still somewhat carefully controlled datasets...

United brings a lot of this on themselves by refusing to publish the info. Used to run a tool for internal United employees that had a support form. We had to turn it off because we got inundated with United customer looking for help with their flights. AI makes that problem worse.

Ai should not be called intelligent. That is beyond wishful thinking. It has no ability to quantify valued information verse low value. Just how much something appears. It’s close to useless if you can’t on accuracy it’s just further diluting and already waining clarity of quality info.

This is such a huge problem that there are entire scam call centers now pretending to be US corpos. Like professional operations with hundreds of agents. Google fucking sucks.

That was happening pre-ai to be fair. I googled “US ESTA fee” from Auckland Airport once to hastily pay for an ESTA fee for my son’s new passport. First result scammed me out of $100 (the fee is $14)

Most e-tickets needed to fly to certain countries are free, as long as you don't need a visa for that trip. But in the rush of the moment (you didn't know the requisite and are already checking in) it's easy to fall for it if it's the first page that pops in your results. I almost did.

I know, I know. Sorry, didn't mean yours was wrong, just that when it happens at the airport it's easy to fall for it because it's a moment of rush and vulnerability.

There's all sorts of variations of this. I've seen them for locksmiths, every government service, NY's MTA's lost and found... Unfortunately at this point the only way to avoid this is to start at the official website and work your way through it: www.cbsnews.com/newyork/news...

cbsnews.comMTA Warns Customers Of Possible Lost And Found ScamThe agency warned not to provide credit card information to third-party websites that solicit unofficial lost property reports.

I’ve been worried for a long time about AI getting stuff wrong just cause it’s not very good, but I hadn’t even considered people gaming the system to *push* bad info into AI summaries (which I’m sure is what happened here). this is truly dangerous shit.

This has happened with people using AI autocompletion for computer code. Not the easiest article to understand if you don’t know about programming but it’s basically literally like putting a non-exist phone no. into all its output, so someone registered that # and all the calls went to them

theregister.comAI bots hallucinate software packages and devs download themSimply look out for libraries imagined by ML and make them real, with actual malicious code. No wait, don't do that

Google's AI told me yesterday that the resistor colors Orange, Brown, Yellow translated to a 1k value at 0.05% tolerance. It's 300k and, without the fourth stripe, it would be 20% tolerance. As it happens, the fourth stripe was gold, which is 5%. Where it got 0.05% is an unsolvable mystery.

(p.s. she didn’t give the bank info of course lol, she hung up immediately, so we’re fine. and she’s generally skeptical of ai though maybe slightly less hateful of it than me. but she didn’t see something *this* bad coming, and honestly, I don’t blame her. it’s wild.)

Especially considering that their monetization of the algorithm has moved the best results to page 2, averaging result 17 according to Cory Doctorow as cited in his enshitification presentations.

Gaming googles search results has been done for a long time, but it seems scanmers all too often end up at the top now. Especially if you Google stuff like tests/reviews in non-english languages, 8/10 first page results are AI translated/generated scams. With just affilite links if you are lucky.

There is a lot of red-teaming and black hat stuff happening right now. We’ve got to either get to attribution of data or have extremely hermetic build & training pipelines.

The trick with that is they technically are. It's just that they crank the levels of the entire ad right to the limit. So while the show has one scene of a guy shouting, every ad is a guy shouting.

AI at this point in time is incapable of understanding (and thus critiquing) the information it assembles. Ultimately, information that is bad is just as valid to it as good information from reliable sources. A network of statistics doesn't make for comprehension.

high value targets. I have a friend who is a middle aged *airline employee* trying to change a reservation and it took him more than 15 minutes to figure out that his call had gone astray...