Google, OpenAI, Anthropic and Microsoft back a robots.txt rule for AI training

[Nicola_67]

Website owners may soon have a simpler way to keep their content out of AI training. Engadget reports that Google, Anthropic, OpenAI and Microsoft have all committed to honouring a new robots.txt directive that lets publishers say whether their content can be used to train generative AI

The signal comes from an Internet Engineering Task Force draft written by Kevin Kelley, Anthropic's technical lead for standards. At the moment, site owners have to write separate rules for each AI company's crawler

The new directive uses one dedicated section that all participating companies respect. Publishers can use Disallow to block training and Allow to permit specific areas, such as a press kit. It does not block crawling altogether and does not remove sites from Google Search, and Google will apply it only to its Google-Extended agent, which gathers training data for Gemini

Anthropic already supports the signal, Microsoft will start on 1 February 2027, Google in early 2027 and OpenAI in the coming months. It does not undo any scraping that has already happened

A single shared rule is a big improvement on the current mess of crawler specific instructions. Publishers have long complained about AI companies using their work without clear permission. Agreement between the biggest labs is a meaningful step, even if it only applies from now on

Will publishers use this to opt out of AI training? And is a robots.txt rule enough to protect creators?

Google joins OpenAI and Anthropic in promising new scraping controls - Engadgetwww.engadget.com