What happened: On September 15, 2026 Cloudflare shipped a new crawler control called Disallow AI Training, and it is on every plan, free included. Turning it on publishes Disallow: Google-Extended and Disallow: Applebot-Extended in your robots.txt, adds a NOARCHIVE signal for Bing, and blocks the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI at the edge. Googlebot, Bingbot and Applebot keep crawling for search. Cloudflare’s reason for splitting the two: “Almost every site owner considers Search beneficial: less than 1% of Cloudflare sites choose to block Search bots.”
Source(s): Cloudflare blog: stay discoverable in search while disallowing AI training and Search Engine Journal, September 15, 2026.
Why Disallow AI Training matters for SaaS sites
Until this week the choice inside Cloudflare was blunt. Block AI Bots stopped mixed-use crawlers outright, and some of those same crawlers are the ones that fetch pages for search results. Plenty of SaaS teams flipped that switch in 2025 to keep their docs and comparison pages out of model training, then spent months wondering why indexing went quiet. Disallow AI Training separates the two questions: do you want to be found, and do you want to be trained on.
The part worth reading twice is the Accountable label. Cloudflare applies it to operators that publish a separate token for training, which right now covers Apple, Google, Microsoft, Amazon, Anthropic, Meta and OpenAI. Accountable operators keep their search access under Disallow AI Training. Training-only crawlers do not. If a company will not split its crawlers, it loses access, which is the pressure Cloudflare is applying.
What Disallow AI Training covers, and what it does not
| Behaviour | Under Disallow AI Training |
|---|---|
| Googlebot crawling for classic search | Allowed |
| Bingbot and Applebot crawling for search | Allowed |
| Google-Extended (Gemini app grounding) | Disallowed in robots.txt |
| Applebot-Extended | Disallowed in robots.txt |
| Amazon, Anthropic, Meta, OpenAI training crawlers | Blocked at the edge |
| AI Overviews and AI Mode appearance | Not affected |
| Bing training signal | NOARCHIVE meta tag, support still partial |
That last row before the Bing one is the one most teams get wrong. AI Overviews and AI Mode are served off the normal Googlebot index, so Disallow AI Training does nothing to them. Those surfaces are controlled from Search Console and from the nosnippet family of tags, not from a Cloudflare toggle. If you have been telling your exec team that blocking AI bots keeps you out of AI Overviews, that has never been true, and it is still not true. Our guide to AI Overviews visibility for SaaS walks through what actually moves there.
The migration matters too. Cloudflare moved existing Block AI Bots and Block on pages with ads customers onto Disallow AI Training automatically. So your robots.txt may have changed on September 15 without anyone on your team touching it, and new ad-monetised domains get the setting by default.
What to do this week
- Pull your own robots.txt and read it. If
Google-ExtendedorApplebot-Extendedappeared there in the last few days, Disallow AI Training is on and the migration did it for you, not a person. - Check what is actually blocked at the edge. In the Cloudflare dashboard go to AI Crawl Control and look at the per-crawler list rather than the summary. A stale Block rule from last year can still sit above the new setting. Then confirm nothing in search is affected with an indexing pass and the URL Inspection tool.
- Decide the training question on business grounds, not fear. If your growth comes from docs, integrations pages and comparison content that assistants cite back to buyers, Disallow AI Training costs you nothing on citations, because citation traffic runs through the search and agent crawlers, not the training ones.
- Log crawler hits by user agent and verify by IP range for a fortnight after any change. Run the sitemap checker at the same time so you can separate a crawl policy problem from a plain sitemap problem.
Quick recap: Disallow AI Training
- Disallow AI Training launched September 15, 2026 on all Cloudflare plans.
- It writes Google-Extended and Applebot-Extended disallows into robots.txt and blocks training-only crawlers at the edge.
- Googlebot, Bingbot and Applebot keep crawling for search under Disallow AI Training, so rankings and indexing are untouched.
- Accountable operators (Apple, Google, Microsoft, Amazon, Anthropic, Meta, OpenAI) keep search access; crawlers that refuse to separate training do not.
- Disallow AI Training has no effect on AI Overviews or AI Mode, which are managed in Search Console.
- Old Block AI Bots settings were migrated over automatically, so audit robots.txt before assuming your config is unchanged. More background in our robots.txt blocking guide and technical SEO for SaaS.
