Post

The greatest danger from AI isn’t that it suddenly “decides to kill us.” I’m increasingly more concerned about the human systems we build around it: bad data, security vulnerabilities, poor oversight, misplaced trust, inadequate safeguards and failures to verify before acting.

Some personnel expected Maven to flag stale or inconsistent intelligence. It didn’t. The site passed through multiple stages of human review before the strike was authorized. Horrifyingly more than 150 people were killed, including at least 123 children.

Humans are putting increasingly powerful technology into systems with bad information, vulnerabilities and imperfect oversight then trusting the technology enough that nobody catches the error before the consequences become irreversible.