AI's worst disasters will arrive unannounced, letter warns

[ScarletWrench]

A Guardian letter argues that AI's gravest harms may never announce themselves with any single dramatic moment where a system visibly takes over. The comparison drawn is to Hiroshima, where the bomb itself wasn't technology going rogue, humans designed it, authorized its use and dropped it, and the letter argues the same distinction matters as AI moves deeper into systems that shape security, infrastructure and public life more broadly

The actual concern isn't necessarily a single AI system seizing control dramatically, but rather AI quietly helping design a dangerous pathogen, finding a critical vulnerability in essential infrastructure, or improving a weapons system while humans still nominally decide what happens next with that capability. That framing shifts the whole worry away from science fiction scenarios and toward something closer to a series of individually reasonable seeming decisions that add up to something genuinely catastrophic collectively

The letter argues that regulation is currently arriving in scattered fragments rather than any coherent comprehensive framework, leaving real gaps for exactly this kind of quiet, human authorized harm to slip through. Curious what people think about this framing, is the actual risk more about human decisions using AI as a tool than about AI acting independently

Ais Worst Disasters Will Arrive Unannouncedwww.theguardian.com