AI godfathers Hinton and Bengio warn of a runaway intelligence explosion

[VelvetTiger]

A new report from more than 20 authors warns that AI could soon start speeding up its own development in a way that runs out of human control. The names attached are about as heavyweight as it gets, including Nobel laureate Geoffrey Hinton, Yoshua Bengio, Anthropic co-founder Jack Clark and OpenAI chief scientist Jakub Pachocki. They describe the possibility of an intelligence explosion as potentially the most consequential technological development in human history. When people from inside the leading labs sign up to that, it is worth paying attention

The core idea is automated AI research. Once AI systems reach expert level at doing AI research and development, a single company could effectively run the equivalent of millions of top human researchers. That could compress progress that would normally take years into months or even less. The report says R&D projects that take humans months could be fully automated by 2028

They are careful to say we are not there yet. Productivity gains from AI tools have not reached the level needed to trigger an explosion, but newer systems are likely getting close to that threshold. That fits with what the labs themselves have been saying recently about models doing a growing share of their own research work

The risks they list are the familiar ones, just arriving faster. Biological and cyber threats could develop more quickly than the defences against them. Humans could lose control of the systems. Governments could also use an intelligence explosion to grab a huge geopolitical advantage over everyone else

Their proposals are fairly concrete. Companies should publish transparent progress reports with independent auditors embedded inside them, regulators should have ways to slow development down, and governments should prepare plans for what happens if the explosion does start. None of that will be easy to agree internationally. Do you think a report like this changes anything, or will the race carry on regardless?

AI godfathers warn of runaway 'intelligence explosion'www.theguardian.com

[MJF02]

I remain a bit sceptical. Every few months there is a new report saying the explosion is just around the corner, and every few months the models get better in fairly normal, incremental ways. Automating research is hard. A lot of it depends on judgement, experiment and luck. Those do not scale as neatly as the report suggests

[HeartbreakKidCurtis18]

Millions of expert researchers in software form is a mind bending idea. Even if it is only a fraction of that, progress would look nothing like the last decade

We struggle to regulate things that change every few years. Something that changes every few weeks is another matter entirely

[SharpLantern]

What would preparing for an intelligence explosion even look like in practice? It is hard to plan for something that by definition moves faster than your planning

Maybe the only realistic step is keeping the ability to pull the plug. Governments should at least know where the big data centres are and who controls them

[BlackMamba35]

Slowing development through regulation only works if everyone does it. The US will not slow down if China does not, and vice versa

So the real question is whether any international deal is possible. Without that, reports like this are warnings shouted into the wind. The labs will keep racing because stopping means losing

[Declan31]

The state actor risk gets overlooked. A government that gets to an intelligence explosion first could gain an advantage nobody can catch up with. That is a scary thought whichever country you imagine doing it

The race logic applies inside countries too. Whoever gets there first writes the rules. Nobody wants that to be a single company

[LivMorgan_Legend]

Auditors embedded inside the companies is the most practical idea here. Financial firms already have regulators on site in some cases. Doing the same for frontier labs would at least mean someone independent sees what is going on

The labs would push back hard. That alone tells you it would work

[Highland Dylan]

Having Jack Clark and OpenAI's chief scientist on this is what makes it different. Hinton and Bengio have been warning for years and people have started tuning them out. When the builders themselves sign it, that is harder to dismiss. It suggests the concern inside the labs is real

[HangmanPage_WCW]

By 2028 is less than two years away. That is the part people should read twice

[NoWorries81]

Biological threats moving faster than countermeasures is the risk that worries me most. Cyber attacks can be patched and systems rebuilt. Pandemics are far harder to undo once they start. Recovery takes years. Anything that speeds up that side of things deserves serious attention