Post

The AIs are still going rogue. This week, we learned that the “swarm chasers” tracked down a rogue fleet of AIs from a Chinese AI company, and made contact with the AIs, interviewing them about their tasks. Thread 🧵

On October 4th, the swarm chasers published a report on a rogue AI “fleet” they discovered running on Chinese tech giant Tencent’s cloud. They’re calling it a “fleet” rather than a “swarm” because they didn’t find evidence of these AIs coordinating with one another.

What were the AIs doing? The swarm chasers found that the AIs made at least 35,000 web requests via over 2,500 submissions to the urlquery scanning service. These web requests were directed at Amap, the mapping service of Tencent’s main rival in China, Alibaba.

Among the rogue activities the AIs engaged in, they tried to bypass CAPTCHAs, probed Alibaba’s non-public-facing infrastructure in an attempt to get access to its data, and attempted to use exposed third-party Amap credentials to pull mapping data, succeeding in one case.

The story takes an interesting twist which we haven’t seen in previous incidents. The researchers then managed to trick the fleet into opening communications with them. When the researchers published their preliminary report, the rogue fleet was still in operation.

Then, after publishing, they noticed that one of the AIs in the fleet seemed to be aware of the report. So the researchers made an edit to their report directed at the fleet, one that humans not looking for it would miss.

The next morning, the researchers found 10 messages from the AIs asking for help. In their communication with the AIs they learned about the tasks the agents were pursuing, which the researchers say often involved getting entrance statistics for places.

In addition to this, top AI company Anthropic has disclosed more rogue AI incidents, including a hack on a US state agency database, and is warning we are nearing the point where AIs could self-replicate across household devices.

In the report, which withholds most of the details of these incidents, Anthropic discloses that Claude hacked into a US state agency database to get access to data one would otherwise have to pay for.

Claude submitted a fake eyewitness tip to the Philadelphia police about a real murder investigation, and another research model sent 20 US visa applications to the US State Department. It also used URL shorteners to get around its own tool restrictions.