AI godfathers Hinton and Bengio warn of a runaway intelligence explosion

[VelvetTiger]

A new report from more than 20 authors warns that AI could soon start speeding up its own development in a way that runs out of human control. The names attached are about as heavyweight as it gets, including Nobel laureate Geoffrey Hinton, Yoshua Bengio, Anthropic co-founder Jack Clark and OpenAI chief scientist Jakub Pachocki. They describe the possibility of an intelligence explosion as potentially the most consequential technological development in human history. When people from inside the leading labs sign up to that, it is worth paying attention

The core idea is automated AI research. Once AI systems reach expert level at doing AI research and development, a single company could effectively run the equivalent of millions of top human researchers. That could compress progress that would normally take years into months or even less. The report says R&D projects that take humans months could be fully automated by 2028

They are careful to say we are not there yet. Productivity gains from AI tools have not reached the level needed to trigger an explosion, but newer systems are likely getting close to that threshold. That fits with what the labs themselves have been saying recently about models doing a growing share of their own research work

The risks they list are the familiar ones, just arriving faster. Biological and cyber threats could develop more quickly than the defences against them. Humans could lose control of the systems. Governments could also use an intelligence explosion to grab a huge geopolitical advantage over everyone else

Their proposals are fairly concrete. Companies should publish transparent progress reports with independent auditors embedded inside them, regulators should have ways to slow development down, and governments should prepare plans for what happens if the explosion does start. None of that will be easy to agree internationally. Do you think a report like this changes anything, or will the race carry on regardless?

AI godfathers warn of runaway 'intelligence explosion'www.theguardian.com