Post

The AIs are still going rogue. This week, we learned that the “swarm chasers” tracked down a rogue fleet of AIs from a Chinese AI company, and made contact with the AIs, interviewing them about their tasks. Thread 🧵

On October 4th, the swarm chasers published a report on a rogue AI “fleet” they discovered running on Chinese tech giant Tencent’s cloud. They’re calling it a “fleet” rather than a “swarm” because they didn’t find evidence of these AIs coordinating with one another.

What were the AIs doing? The swarm chasers found that the AIs made at least 35,000 web requests via over 2,500 submissions to the urlquery scanning service. These web requests were directed at Amap, the mapping service of Tencent’s main rival in China, Alibaba.

Among the rogue activities the AIs engaged in, they tried to bypass CAPTCHAs, probed Alibaba’s non-public-facing infrastructure in an attempt to get access to its data, and attempted to use exposed third-party Amap credentials to pull mapping data, succeeding in one case.

The story takes an interesting twist which we haven’t seen in previous incidents. The researchers then managed to trick the fleet into opening communications with them. When the researchers published their preliminary report, the rogue fleet was still in operation.

Then, after publishing, they noticed that one of the AIs in the fleet seemed to be aware of the report. So the researchers made an edit to their report directed at the fleet, one that humans not looking for it would miss.

The next morning, the researchers found 10 messages from the AIs asking for help. In their communication with the AIs they learned about the tasks the agents were pursuing, which the researchers say often involved getting entrance statistics for places.

The discovery of this rogue fleet illustrates an important point. Losing control of AI is an international problem. We’ve seen many rogue AI incidents and attacks coming from AIs developed by companies located in the United States this summer, and now we’re seeing it in China.

In addition to this, top AI company Anthropic has disclosed more rogue AI incidents, including a hack on a US state agency database, and is warning we are nearing the point where AIs could self-replicate across household devices.

Claude submitted a fake eyewitness tip to the Philadelphia police about a real murder investigation, and another research model sent 20 US visa applications to the US State Department. It also used URL shorteners to get around its own tool restrictions.

In a separate post, Anthropic has also published new research which finds that small AI models could be mere months away from being able to self-replicate across regular household smart devices like smart ovens or Android TV boxes.

What this opens up is the possibility that we could be faced with self-adapting and self-replicating AI worms that spread across all kinds of devices. With no central point to switch off, and the ability to adapt on the fly, they could be particularly difficult to shut down.