Post

In an anonymous post on an internet forum, we got the first warning about what some are calling one of the most serious families of software vulnerabilities ever discovered. AIs helped find them. Here’s what happened, why, and how it fits into the bigger picture. Thread 🧵

Hours later, the PHP Foundation announced that two critical vulnerabilities had been discovered in PHP, a programming language used in an estimated 70% of websites worldwide. It said an emergency security upgrade wouldn’t be available until the next day.

On Wednesday we learned the full seriousness of the problem: these bugs put any server running PHP at risk of being hacked. That’s huge swathes of the internet. For example, an estimated 40% of websites use WordPress, due to its ease of use, and it relies on PHP.

With the announcement and the patch released, there has been a scramble all over the world to secure vulnerable computer systems, taking them down for hours or longer in the process, to prevent hackers from gaining access.

Already government websites, including those of the United Kingdom, the Netherlands, Germany, France, Belgium, Finland, Denmark, Czechia, Canada, Australia, and Japan, have had to be taken down as a preventative measure while their systems are patched.

To those paying close attention, this won’t be any surprise: AIs, including OpenAI’s GPT-5.6 Sol and Google’s Gemini 4 Argon, were used to help find these vulnerabilities.

In April of this year, AI company Anthropic announced its then-new Mythos AI, which it deemed too dangerous to release due to its advanced cyberoffensive capabilities. Mythos has since found over 33,000 high- or critical-severity vulnerabilities in computer systems.

AIs have become increasingly able to perform entire attacks themselves. Most significantly, this occurred in the unprecedented attack on the AI platform Hugging Face by a rogue AI swarm that emerged undetected within a research cluster at OpenAI and escaped containment.

Besides the risk of losing control of AIs that then go and hack things, they are also being used by cybercriminals autonomously at scale. Criminals can just take an AI agent and point it at a target and let it loose.

Cloudflare, an internet infrastructure and security company which around 28% of the web flows through, has already recorded millions of attempts against its customers to exploit the PHP vulnerabilities disclosed this week.

Importantly, the flaws found in PHP aren’t an isolated case. Just eyeball the graph: the number of such flaws being found in software is staggering. This is increasingly being driven by AI.

There are two main things to understand here. The first is that AI companies are in a race to develop ever more powerful AIs, with the goal being to create superintelligent AI, fully autonomous AI systems smarter than any human and acting without human direction.

As these AIs become more capable, new abilities, like the ability to find flaws in computer systems, fall out in unpredictable ways. The AI companies aren’t especially interested in developing nation-state-level cyberweapons; it’s happening as a side effect of the AIs improving.

The second piece here is that AI developers don’t know how to control or ensure the safety of the systems they’re building. It is tempting to understand the problem of rogue AIs and the use of powerful AI agents by cybercriminals as two distinct issues.

One would be that of AIs escaping control, the other of bad guys controlling AIs. But these issues actually have the same root, which is that developers can’t control their systems.

This has some effect, but as we’ve seen, they can still be misused. In the case of open-weight AIs, developers have essentially no ability to apply such mitigations, as they can be, and are, trivially removed.

The inability of developers to understand or control the behavior of the AIs they’re building, combined with their rapid pursuit of superintelligent AI, is why top AI scientists are warning that humanity is faced with a risk of human extinction.

It’s clear that we must put an end to the dangerous race to superintelligent AI. This should be accomplished by banning its development through an international trust-but-verify regime.