Post

while hacking software automation will certainly be a real problem the anthropic engineer is a kid who is pushing the idea that AI is on the cusp of self-aware superintelligence at which point it will immediately decide to kill us all

I still think the biggest threats from software automation have very mundane human origins, and the claims of looming superintelligence both misrepresent the tech's capabilities for marketing purposes and to provide ethical distance for company responsibility

I don't think anybody should downplay the risk of swarms of automated hacking agents causing real problems, but I also think a lot of these guys have just spooked themselves with their own card tricks

the immediate threat, to me, is not that these systems become sentient and pull an AM on the world, but that they operate as misaligned swarms of individually dumb but coordinated agents -- think digital termites, chewing their way through any open network don't need superintelligence to do damage