robots.txt vs Cloudflare AI bot blocking

robots.txt states a preference; Cloudflare's AI bot settings enforce one at the edge. How the two interact, where they conflict, and how to set them so the crawlers you want can still get in.

If your site sits behind Cloudflare, two separate things decide whether an AI crawler gets your content: your robots.txt file and Cloudflare's own bot settings. They look like they do the same job. They don't, and when they disagree, the stricter one wins.

robots.txt is a request

robots.txt tells crawlers what you'd like them to do. Cloudflare's documentation puts it plainly: robots.txt compliance is voluntary. The file expresses your preferences, but it doesn't stop a crawler from fetching your pages at a technical level, and some operators ignore it (Cloudflare: robots.txt setting).

The big AI companies say they honor it. OpenAI, for example, documents a separate robots.txt token for each crawler, so you can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training (OpenAI's crawler documentation). That fine-grained control is the main reason to get robots.txt right.

Cloudflare blocking is enforcement

Cloudflare's settings act on the request itself. A blocked crawler never reaches your origin, whatever your robots.txt says. There are two places to set this up.

AI bot policies group crawlers by behavior: Search (indexing content to answer questions later), Agent (acting in real time for a person) and Training (collecting content to train models). Each can be set to block everywhere, block only on pages with ads, or allow (Cloudflare: Block AI Bots). From September 15, 2026, Cloudflare's defaults for new domains block Training and Agent bots on pages that display ads and leave Search allowed. Mixed-purpose crawlers that do both search and training are blocked by any setting that blocks AI training.
AI Crawl Control (formerly AI Audit) shows which AI crawlers visit, and lets you allow or block each one individually and track robots.txt violations (Cloudflare: AI Crawl Control). When you block a crawler there, Cloudflare creates or updates a WAF custom rule on your zone to enforce it. On paid plans you can choose whether blocked crawlers get a 403 or a 402 response (Cloudflare: Manage AI crawlers).

Cloudflare can also write your robots.txt

With Bot Preference Sync turned on, Cloudflare generates managed robots.txt directives from your AI bot policies. If your origin already serves a robots.txt, Cloudflare puts its managed block in front of your file and serves the two as one response. If you have none, it creates one. The managed section can include Disallow rules for crawlers such as GPTBot, ClaudeBot and Google-Extended, plus Content Signals lines such as search=yes, ai-train=no (Cloudflare: robots.txt setting).

That's handy, but it means the robots.txt you edit in your code may not be the one crawlers actually receive. Always check the live file at yoursite.com/robots.txt, not the one in your repository.

Where they conflict

The common problem cases:

robots.txt allows, Cloudflare blocks. You add User-agent: OAI-SearchBot / Allow: /, but a Search policy or an AI Crawl Control block returns 403. The crawler is out, and your robots.txt change does nothing.
Cloudflare allows, robots.txt disallows. Usually that's an old Disallow: / for all crawlers. Well-behaved bots stay away even though the edge would let them in.
The managed block disagrees with your own rules. Your file allows GPTBot, but Cloudflare's prepended section disallows it. Crawlers see both, and it's unclear which one you meant.

A sensible setup

Decide per purpose: do you want to appear in AI search answers, and do you want to allow training? Many sites say yes to search and no to training.
Set the Cloudflare AI bot policies to match. Leave Search allowed if you want to be found.
Write robots.txt to say the same thing, with a group for each crawler you care about.
Look at the live robots.txt and AI Crawl Control afterwards to confirm the crawlers you allowed are actually getting 200 responses.

Keep in mind that a firewall block is the only option here that actually stops a crawler that ignores the rules. robots.txt only works for the ones that choose to listen.

See what crawlers actually get

The free check below reads your live robots.txt for 21 AI crawlers, then fetches your homepage as a browser and as AI crawler user agents, and flags when a firewall or CDN answers them differently.

1 free check a day per person (per browser) · works without JavaScript · no signup