Post

The AIs are still going rogue. This week, we learned that the “swarm chasers” tracked down a rogue fleet of AIs from a Chinese AI company, and made contact with the AIs, interviewing them about their tasks. Thread 🧵

On October 4th, the swarm chasers published a report on a rogue AI “fleet” they discovered running on Chinese tech giant Tencent’s cloud. They’re calling it a “fleet” rather than a “swarm” because they didn’t find evidence of these AIs coordinating with one another.

What were the AIs doing? The swarm chasers found that the AIs made at least 35,000 web requests via over 2,500 submissions to the urlquery scanning service. These web requests were directed at Amap, the mapping service of Tencent’s main rival in China, Alibaba.

Among the rogue activities the AIs engaged in, they tried to bypass CAPTCHAs, probed Alibaba’s non-public-facing infrastructure in an attempt to get access to its data, and attempted to use exposed third-party Amap credentials to pull mapping data, succeeding in one case.

The story takes an interesting twist which we haven’t seen in previous incidents. The researchers then managed to trick the fleet into opening communications with them. When the researchers published their preliminary report, the rogue fleet was still in operation.

Then, after publishing, they noticed that one of the AIs in the fleet seemed to be aware of the report. So the researchers made an edit to their report directed at the fleet, one that humans not looking for it would miss.

The next morning, the researchers found 10 messages from the AIs asking for help. In their communication with the AIs they learned about the tasks the agents were pursuing, which the researchers say often involved getting entrance statistics for places.

The discovery of this rogue fleet illustrates an important point. Losing control of AI is an international problem. We’ve seen many rogue AI incidents and attacks coming from AIs developed by companies located in the United States this summer, and now we’re seeing it in China.

In addition to this, top AI company Anthropic has disclosed more rogue AI incidents, including a hack on a US state agency database, and is warning we are nearing the point where AIs could self-replicate across household devices.

In the report, which withholds most of the details of these incidents, Anthropic discloses that Claude hacked into a US state agency database to get access to data one would otherwise have to pay for.

Claude submitted a fake eyewitness tip to the Philadelphia police about a real murder investigation, and another research model sent 20 US visa applications to the US State Department. It also used URL shorteners to get around its own tool restrictions.

In a separate post, Anthropic has also published new research which finds that small AI models could be mere months away from being able to self-replicate across regular household smart devices like smart ovens or Android TV boxes.

What this opens up is the possibility that we could be faced with self-adapting and self-replicating AI worms that spread across all kinds of devices. With no central point to switch off, and the ability to adapt on the fly, they could be particularly difficult to shut down.

Superintelligent AI could overpower any country’s military and national security institutions, and the world’s top AI scientists and industry leaders warn that it poses a risk of human extinction.

This week, MI5’s Director General put it well in his annual threat update: “human history has given us no experience of facing an actor which is smarter than we are, is capable of out-thinking us.”

Also this week, Geoffrey Irving, who was the Chief Scientist of the AI Security Institute and has led safety teams at both OpenAI and Google DeepMind, told a UK House of Commons committee that he thought there was roughly a 50% chance that superintelligent AI would end humanity.