Is the AI doom talk is oddly convenient for the companies selling the AI?

[Blue Coder]

The Guardian's Wednesday briefing takes a step back from the usual extinction headlines and asks a blunter question. Who actually benefits when a chief executive stands up and says his own product might wipe out humanity. The answer the piece lands on is uncomfortable, because heavy regulation built around fear of a runaway superintelligence tends to favour the handful of firms who can already afford the compliance burden, while smaller labs and open source developers get squeezed out entirely.

The article leans on a familiar cast to make the point. Eliezer Yudkowsky is quoted warning that a sufficiently capable system could kill everyone simply because it does not particularly want us around, and Max Tegmark's long running campaign for a research pause gets a mention too. Against that sits a set of skeptics who think the framing is self serving rather than sincere. Andrew Ng calls licensing requirements a massively, colossally dumb idea, Yann LeCun accuses the doomers of risking a real catastrophe by slowing down beneficial progress, and Arvind Narayanan argues that since capabilities only emerge from training data, the extinction scenario is overblown in the first place.

Garry Tan's contribution is the sharpest dig in the whole piece. He points out that regulation built around fear only really helps the largest firms, the ones with the legal teams and lobbyists to navigate whatever licensing regime eventually gets written. Smaller competitors do not get that luxury. When OpenAI, Google DeepMind and Anthropic leadership all sign a joint statement putting AI extinction risk on the same footing as nuclear war and pandemics, the briefing suggests it is worth asking whether that statement also happens to be a moat.

None of this means the underlying safety concerns are fake, and the briefing is careful not to claim that. The argument is narrower and in some ways more cynical, which is that sincere fear and convenient self interest can coexist inside the same press release. A lab can genuinely believe its own technology is dangerous while also recognising that saying so out loud, loudly and often, happens to keep the regulatory ladder pulled up behind it.

What the piece does not fully resolve is what a better alternative regulatory posture would even look like, one that takes catastrophic risk seriously without simply anointing the incumbents as the only safe custodians of the technology. That tension between wanting oversight and worrying about who gets to write the rules has been running through the AI safety debate all year, from the King Charles summit to the SB53 disputes, and this briefing is really just the latest and most explicit statement of the suspicion that has been building underneath it.

Wednesday briefing: Why tech companies might be only too happy for us to believe AI will 'kill us all'www.theguardian.com

[Di82]

Garry Tan's quote is the one that sticks with me because he has no reason to defend the doomers and no reason to defend big labs either, he is just pointing at the mechanism. Licensing regimes cost money to comply with. Whoever has the most money wins by default regardless of how the rule was originally justified.

[Odd Voyager]

Narayanan's point about capabilities only emerging from training data is doing a lot of work in this piece and I am not sure it holds up as well as the article implies. Emergent behaviour from scale has surprised plenty of researchers before, dismissing extinction risk because of one theoretical framing feels premature either way.

[ChrisJericho92]

I think this framing lets the smaller AI companies off the hook a bit too easily though. Plenty of startups are also raising money on the promise that their model will be transformative and dangerous, it is not just OpenAI and Anthropic doing this. Fear sells at every size of company, it is just that only the big ones can afford to survive the regulation that follows.

[PlanckLimit65]

Andrew Ng calling licensing colossally dumb is rich coming from someone who has spent his whole career telling people AI will change everything. You cannot have it both ways, either the technology is transformative enough to need careful handling or it is not.

[Cass93]

The bit that gets skipped in this kind of piece is that regulators are not stupid either. Plenty of policy people have read the exact same critique Tan is making and are trying to write rules that scale with company size rather than imposing a flat compliance cost. Whether they succeed is another question but the framing that regulation always favours incumbents is not automatically true.

[Ruby92]

This is the same argument people made about crypto exchanges lobbying for regulation a few years back. Once you are big enough the rules become a wall that keeps everyone smaller out, and you get to look responsible while doing it. I am not saying every doomer is lying, some of them clearly believe what they are saying, but the incentive alignment is too neat to ignore

[Omega16]

What strikes me is how long this argument has been sitting there in plain sight and how few outlets were willing to say it plainly until now. Everyone quietly suspected the joint safety statements were partly strategic. It took this long for a mainstream piece to actually say it out loud instead of hedging around it