AI crawlers blocked at the edge

What this check does

Requests your homepage twice, identifying with the exact user-agent strings OpenAI and Anthropic publish for GPTBot and ClaudeBot. The check fails when either request gets HTTP 401, 403 or 429, or a 503 challenge page from Cloudflare.

When the response carries Cloudflare headers the finding names Cloudflare; otherwise it says “WAF/CDN” so you know to look at your firewall or security plugin.

This is a major finding: it overrides everything else in the AI-readability module. An Allow: / in robots.txt is meaningless if the crawler is turned away before it can read the file.

Why it matters

robots.txt is a request. A 403 is a wall.

Well-behaved AI crawlers read robots.txt first and obey it. But a firewall rule that matches the user agent answers before robots.txt is served, so the crawler never sees your permissions. From the crawler’s side the site is simply down.

The most common cause in 2026 is Cloudflare AI Crawl Control (formerly “Bots”). Since mid-2025 Cloudflare blocks known AI crawlers by default on new zones, and many site owners switched it on without realising it also removes them from ChatGPT, Claude and Perplexity citations. Other causes:

• Cloudflare Bot Fight Mode / Super Bot Fight Mode
• Custom WAF rules matching GPTBot, ClaudeBot or the word bot
• WordPress security plugins (Wordfence, Shield Security, All In One WP Security) with “block AI bots” toggles
• nginx / Apache user-agent rules copied from a “block AI scrapers” snippet

If blocking AI crawlers is a deliberate decision, this finding is confirming that it works. If you want to be cited by answer engines, it is the first thing to fix.

How to fix it

Cloudflare

1. Open your domain in the Cloudflare dashboard and go to AI Crawl Control (older accounts: Security → Bots).
2. Turn off Block AI bots, or keep it on and set GPTBot and ClaudeBot to Allow in the crawler list.
3. If you also use Bot Fight Mode, check that it isn’t challenging them. Super Bot Fight Mode lets you allow “verified bots”; both crawlers are on Cloudflare’s verified list.

WordPress security plugins

Look for a “Block AI bots”, “Block AI scrapers” or “Bad bots” setting and remove GPTBot and ClaudeBot from the list.

nginx / Apache

# Remove or narrow rules like this one
if ($http_user_agent ~* (GPTBot|ClaudeBot)) {
  return 403;
}

Verify

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" https://yourdomain.com/ | head -1
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://yourdomain.com/ | head -1

Both should print HTTP/2 200. Then re-run the audit.

Frequently asked questions

My robots.txt allows GPTBot. Why does the audit still fail?

Because the block happens one layer earlier. The crawler asks for a page, the firewall answers 403, and robots.txt is never read. Fix the firewall first; the robots.txt rule is checked separately in Allow AI crawlers.

Does this also affect Googlebot?

No. Cloudflare and most plugins treat Googlebot and Bingbot as verified search crawlers and leave them alone. This check only probes the two main AI crawlers.

Can I allow ChatGPT citations but block training?

Partly. OpenAI uses GPTBot for training and OAI-SearchBot / ChatGPT-User for search and browsing. Allow the latter two at the edge and keep GPTBot blocked in robots.txt if that is your intent. Anthropic uses ClaudeBot for both purposes.