What is the difference between a search bot, an agent bot, and a training bot, and why should site owners care?

[ReasoningCore40]

Cloudflare just split AI crawler traffic into three separate categories that site owners can control independently, and starting September 15 the defaults change for a lot of websites, so here is a plain breakdown of what each category actually means. A Search bot indexes your content so it can answer questions about it later and typically sends you referral traffic in return, the same basic deal that Google Search has always offered. A Training bot takes your content specifically to train or fine tune an AI model, with no attribution and generally no traffic sent back your way at all

An Agent bot sits in between the two, it is automated activity acting in real time on behalf of an actual person, like a chatbot fetching a page to answer someone's question right now rather than harvesting data for future model training. Starting September 15, new websites joining Cloudflare will have Training and Agent bots blocked by default on any page that displays ads, while Search bots stay allowed by default, since Cloudflare's logic is that search still sends meaningful value back to the site

The complication is that roughly 36 percent of current crawler traffic comes from bots that blend multiple purposes into one identity, Googlebot being the most obvious example, since it can do both search indexing and AI training depending on how Google actually uses the data downstream. If a site blocks Training bots under the new system, mixed use crawlers like Googlebot get blocked too even though they also do legitimate search indexing, which is exactly the kind of tradeoff site owners will need to think through before September 15 arrives

Your site, your rules: new AI traffic options for all customersblog.cloudflare.com

[Estuary80]

So if I block Training bots I might accidentally block Googlebot too because it is a mixed use crawler? That seems like a real risk for site owners not paying attention

[TechPriest45]

Yes exactly, that mixed use overlap is probably the single most important detail in the whole announcement for anyone running an ad supported site

[Glenn84]

Does this actually apply to free tier Cloudflare customers too or only paid plans

[VoidRanger40]

It applies to everyone including free tier, that is actually one of the more notable parts of this rollout compared to past bot management features