ai

Cloudflare's managed robots.txt blocks AI crawlers

Cloudflare can inject its own robots.txt block above your rules. MetricSpot checks whether that managed block disallows GPTBot or ClaudeBot, even when your own file allows them.

What this check does

Looks for the markers Cloudflare wraps around the directives it injects into /robots.txt:

# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow: /
# END Cloudflare Managed Content

# Your own rules start here
User-agent: GPTBot
Allow: /

The managed section is judged on its own. The check fails when it contains Disallow: / for GPTBot or ClaudeBot. Your own rules below the marker are evaluated separately by Allow AI crawlers, so the two findings tell you exactly which layer is blocking.

The check is only emitted when the markers are present. Sites that don’t use Cloudflare’s managed robots.txt never see it.

Why it matters

Cloudflare’s “Manage AI bots’ robots.txt” option (under AI Crawl Control) rewrites the file your origin serves. It prepends a block that:

  • declares Content-Signal: ai-train=no for every crawler, an opt-out under the EU copyright directive
  • adds Disallow: / for GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Amazonbot and others

Site owners often turn it on for the Content Signal and don’t notice the Disallow lines. Then they add Allow: / further down and assume it wins. It doesn’t, reliably: robots.txt parsers merge groups differently, several crawlers take the first matching group, and Cloudflare usually also blocks the same bots at the edge (see AI crawlers blocked at the edge).

The result is a robots.txt that says one thing at the top and the opposite at the bottom. This check makes the top half visible.

How to fix it

  1. Open your domain in the Cloudflare dashboard and go to AI Crawl Control (older accounts: Security → Bots).
  2. Turn off Manage AI bots’ robots.txt. The injected block disappears within a few minutes and your origin’s file is served as-is.
  3. If you want to keep the Content-Signal line, leave the option on and set GPTBot and ClaudeBot to Allow in the crawler list. Cloudflare then drops their Disallow lines from the managed block.
  4. Check the edge as well: the same screen controls whether those crawlers get a 403 before they ever read robots.txt.

Verify

curl -s https://yourdomain.com/robots.txt | sed -n '/BEGIN Cloudflare/,/END Cloudflare/p'

If the block is gone, or no longer lists GPTBot and ClaudeBot under Disallow: /, re-run the audit.

Frequently asked questions

I never edited robots.txt. Where did this block come from?

From Cloudflare, not your CMS. Anything between # BEGIN Cloudflare Managed content and # END Cloudflare Managed Content is generated at the edge and never touches your server’s file.

Does Allow: / further down override it?

Not dependably. RFC 9309 says an equivalent Allow and Disallow should resolve to Allow when groups are merged, but not every crawler merges groups, and the Cloudflare block sits first. Remove the conflict instead of relying on parser behaviour.

Is the Content-Signal line harmful?

No. ai-train=no is a rights reservation, not a block. You can keep it and still allow GPTBot and ClaudeBot to crawl for search and citations.

Sources

Last updated 2026-09-04