Post

The greatest danger from AI isn’t that it suddenly “decides to kill us.” I’m increasingly more concerned about the human systems we build around it: bad data, security vulnerabilities, poor oversight, misplaced trust, inadequate safeguards and failures to verify before acting.

Minab is a horrifying example. The Iranian school site struck by the U.S. was classified in a military database as an IRGC facility, despite evidence that its use had changed. An analyst documented changes as early as 2019 but in a system that wasn’t connected to the primary targeting database.

Some personnel expected Maven to flag stale or inconsistent intelligence. It didn’t. The site passed through multiple stages of human review before the strike was authorized. Horrifyingly more than 150 people were killed, including at least 123 children.

Humans are putting increasingly powerful technology into systems with bad information, vulnerabilities and imperfect oversight then trusting the technology enough that nobody catches the error before the consequences become irreversible.