How to audit your robots.txt (and CDN) for AI crawlers

The user-agents that matter

• OAI-SearchBot — the crawler behind ChatGPT Search results; GPTBot — content that may be used to train OpenAI's models, a separate opt-in; ChatGPT-User — fetches a page because a user asked, during a conversation.
• Claude-SearchBot — Anthropic's search crawler; ClaudeBot — training collection; Claude-User — user-directed retrieval. Three tokens, three different decisions.
• PerplexityBot — Perplexity's search crawler (Perplexity states it is not used for foundation-model training); Perplexity-User — user-requested fetches, which generally do not follow robots.txt.
• Google-Extended — controls Gemini/AI use of your content (separate from Googlebot: blocking it does NOT affect your search rankings).
• Googlebot — the crawler AI Overviews and AI Mode depend on, since those run on ordinary Search eligibility; Google-Extended is a separate token for Gemini/Vertex use and does not affect Search inclusion.
• Applebot-Extended, CCBot (Common Crawl), Bytespider and friends — training-side, but Common Crawl feeds many models' training data.

Note the pattern: OpenAI, Anthropic and Google each publish separate tokens for training and for search, and document them as independent choices — so you can allow the search crawler that fetches pages for answers while disallowing the training one, or any combination. Read the provider's own bot page before deciding; the roles differ, and they change. The point is choosing deliberately rather than by accident.

The trap: your CDN blocks before robots.txt speaks

robots.txt is a polite request that well-behaved crawlers honor. CDN bot protection is a wall that stops requests before your server answers at all — and it's where most accidental blocking happens. Cloudflare, for example, has offered one-click AI-crawler blocking and has moved toward blocking AI bots by default for new sites; similar toggles exist at other providers and in WAF products.

• Check your CDN dashboard for AI/bot settings explicitly (Cloudflare: the AI-crawler controls; others: bot-fight/WAF rules).
• A perfect robots.txt means nothing if the CDN returns 403 to OAI-SearchBot — and your robots.txt won't show any evidence of it. (A 403 to GPTBot is a different test: it tells you about training access, not about whether ChatGPT Search can retrieve the page.)
• Managed platforms (some site builders, some hosts) ship their own defaults — if you don't remember choosing, verify.

The 15-minute audit

• 1. Read yourdomain.com/robots.txt top to bottom; list every AI user-agent that's disallowed.
• 2. Fetch your homepage as a SEARCH crawler (curl -A "OAI-SearchBot" https://yourdomain.com) — that is the access your citations depend on; test a training agent separately only if you care about that lane — a 403/1020 here with a 200 for normal browsers means the CDN wall, not robots.txt.
• 3. Grep a recent server-log sample for the search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot. Presence = end-to-end access works. Absence over weeks = something upstream is off (or your site is too new/small to be crawled yet — check back).
• 4. Write down what you WANT per bot class (train vs live-answer), and make robots.txt + CDN match it.
• 5. Re-verify after any CDN, host, or plan change — defaults shift under you.

Promvia's GEO audit runs the robots.txt check automatically, and its crawler tracking shows which AI bots actually reach you (verified against published IP ranges, not just user-agent strings — those are trivially faked). But steps 2 and 4 are judgment calls that stay yours.

Should you block instead?

Blocking is a legitimate choice for content businesses whose text IS the product. It's a bad accident for everyone else: a blocked crawler cannot quote a page it never fetched, so the answer is assembled from the pages it can read. The honest framing is a trade — content protection versus AI visibility — and the only wrong outcome is not knowing which side of it you're on.

Frequently asked questions