Blanket AI blocks in robots.txt or your WAF can erase ChatGPT and Claude citations. Split training vs search bots and run a one-hour access checklist.
You cannot be cited from a page the search bot never fetched. In 2026 that failure is often accidental: a blanket "block AI" robots.txt or a CDN/WAF rule that treats assistants like scrapers.
This is the access friction root cause from Why Your Brand Goes Missing.
Training bot ≠ search bot
OpenAI documents separate agents: GPTBot (training), OAI-SearchBot (ChatGPT search indexing), and user-triggered fetchers. Disallowing GPTBot opts content out of training; blocking OAI-SearchBot can keep you out of ChatGPT search answers.[1][2]
Anthropic similarly splits ClaudeBot (training) from Claude-SearchBot / Claude-User. Blocking ClaudeBot does not automatically block search indexing; blocking Claude-SearchBot may reduce search visibility.[3][4]
Exact user-agent strings change — copy from vendor docs when you edit robots.txt.
robots.txt pattern (illustrative)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Confirm against current OpenAI / Anthropic documentation before shipping — strings and guidance evolve.
The second gate: CDN / WAF
Many sites "allow" bots in robots.txt and still return 403/Challenge from Cloudflare or similar bot management. Fetch logs and vendor IP lists. If search bots cannot get 200 HTML, you are invisible for live citation even with perfect copy.[5]
1-hour access checklist
- Open
/robots.txt— find blanketDisallowfor GPTBot / Claude / "AI" groups. - Split training vs search agents per vendor docs.
- Hit money URLs with a known AI search UA (or vendor fetch tools) and confirm 200.
- Review WAF bot scores / managed rules for false positives.
- Soft-404 and JS-only shells still fail extraction — fix with answer blocks.
- Re-test prompts after ~24h (OpenAI notes robots changes can take about a day to propagate for search).[1]
How Attensira fits
Access friction is a diagnose-and-ship problem: find the blocked path, write the robots/WAF/content fix, open a PR. Not a crawler vanity dashboard.
FAQ
If we block all AI bots, can we still rank in Google?
Often yes for classic SEO — and still lose assistant search citations. Decide per product surface.
Do user-initiated fetchers always honor robots.txt?
Vendors differ. OpenAI notes ChatGPT-User is user-triggered and not the same as automated crawl policy. Read current docs; Unknown for edge cases.
Let the search bots read the pages you want cited. Then ship the content fixes. Attensira.
