How AI assistants find your pages

Which crawlers do AI assistants use to find my pages?

Different ones for different jobs, and the split is published rather than inferred. Each major operator runs a separate agent for training, for building a search index, and for fetching a page because a person just asked about it. They carry different robots.txt tokens, and blocking one has no effect on the others. That is the design: Anthropic says outright that it uses different robots "to enable website owner transparency and choice."[^anthropic-purpose-of-split]

Everything else in this chapter follows from that one fact.

The agents, by operator

OpenAI

OpenAI documents four user agents.

If you want to appear in ChatGPT's search answers, OAI-SearchBot is the agent that matters. GPTBot is the training crawler, and the two decisions are independent. A site that disallows GPTBot and allows OAI-SearchBot has declined to be training data while remaining eligible to be surfaced - a position plenty of publishers hold deliberately.

Anthropic

Anthropic documents three robots, and unusually, documents the consequence of disabling each one.

The third column is worth reading twice, because it is the operator itself stating the visibility cost of a block - not a vendor inferring one.

Perplexity

Perplexity documents two agents, and its framing is the most explicit of the three about what each is not for. PerplexityBot "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."[^perplexity-bot-purpose] Perplexity-User handles user-initiated visits, and carries the one behavioural exception every technical reader should know: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."[^perplexity-user-robots]

That sentence is the clearest published statement that a robots.txt disallow is not a universal off switch. It governs automated crawling. It does not necessarily govern a fetch a human just triggered by asking a question.

Perplexity also states the operational details most operators leave you to guess: each setting works independently, and changes may take up to 24 hours to be reflected.[^perplexity-propagation] And it recommends, in its own documentation, allowing PerplexityBot and permitting its published IP ranges if you want to appear in results.[^perplexity-recommendation]

Google

Google is the exception, deliberately. It does not run a separate crawler for AI Overviews or AI Mode: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search."[^google-robots-control] There is no way to be in Google's classic results and out of its AI features by crawler choice, because there is only one crawler. What exists instead is snippet-level control: nosnippet, data-nosnippet, max-snippet and noindex.[^google-snippet-controls] Google points to Google-Extended for limiting AI training and grounding in some of its other systems - which is a different question from AI features in Search.

The practical upshot: on Google, opting out of AI features means accepting a reduced or absent snippet, and a page with no snippet is a page with much less to extract. There is no free version of that trade.

What this means for robots.txt

Three patterns cover most real intentions.

Pattern one - visible everywhere, not training data. The common publisher position. Allow the search and user agents, disallow the training crawlers.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Pattern two - open. No AI-specific rules at all. This is what most sites should do, and most sites that have "an AI crawler policy" would have been better off with it.

Pattern three - closed. User-agent: * with a blanket disallow. This is a legitimate choice and it removes you from AI answers. What makes it a mistake in practice is doing it by accident - which is exactly what happens when a security plugin, a CDN bot-fighting rule, or a copied-from-a-blog robots.txt lands in production without anyone mapping it to the table above.

One thing not to do: block by IP. Anthropic states plainly that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."[^anthropic-ip-blocking-warning] Blocking the request that reads your preferences is not a way to express a preference.

The layer robots.txt does not reach

A crawler can be allowed in robots.txt and still fail to fetch your page, and this is the single most common cause of unexplained invisibility. The block is usually somewhere else:

• A WAF or bot-management rule. Perplexity's own documentation includes

Cloudflare and AWS WAF configuration steps precisely because this happens routinely. A managed "block AI bots" ruleset is one toggle away in most CDN dashboards, and nobody who flipped it remembers doing so.

• Client-side rendering. If the content only exists after JavaScript runs, a

fetcher that does not execute JavaScript sees an empty shell. Google's guidance for AI features asks site owners to make sure important content is available in textual form; that instruction is not decorative.

• Interstitials and consent walls. A cookie banner that gates content

server-side returns the banner to the crawler.

• Rate limiting. A crawler that gets 429s learns to visit less often.

None of these show up in a rank tracker or a content audit. They show up in server logs, which is why crawler log analysis is a distinct capability rather than a report tab, and why the honest first step of any AI-visibility programme is checking whether the fetch is happening at all.

The third category nobody plans for

Training crawlers and search crawlers both run on someone else's schedule. The third kind - ChatGPT-User, Claude-User, Perplexity-User - runs on your prospect's schedule, in the middle of an evaluation, because they typed your name into an assistant and asked what you do.

That changes what a block costs. Blocking a training crawler forgoes a licensing position. Blocking a user agent means that when a buyer asks an assistant a direct question about your product, the assistant cannot read your answer to it and falls back to whatever third-party page it already has. Anthropic states this consequence explicitly for Claude-User.[^anthropic-claude-user] Perplexity goes further and says its user-initiated fetcher generally ignores robots.txt anyway,[^perplexity-user-robots] which means the same intention produces different outcomes on different platforms - a good reason to decide per agent rather than per fashion.

There is a positive version of this too. These agents fetch pages on demand, so the page a buyer's assistant reads is the page you shipped this morning, not the one in an index refreshed last quarter. Documentation, pricing pages and comparison pages are the ones that get fetched this way, and they are the ones worth keeping accurate and readable without JavaScript.

OAI-AdsBot is the odd one out and is not a visibility question at all: OpenAI documents it as validating the safety of pages submitted as ads on ChatGPT.[^openai-adsbot-purpose] If you do not buy ChatGPT ads, it has nothing to do with you; if you do, blocking it breaks your own ad review.

How to verify, in one afternoon

1. Fetch your own robots.txt and read it against the table above. Name each

agent explicitly rather than relying on *, because the consequences differ per agent and a wildcard hides that from whoever reads the file next.

1. Grep your access logs for OAI-SearchBot, GPTBot, ChatGPT-User,

ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. Count requests and status codes per agent for the last 30 days. A 403 or 429 concentration is your answer.

1. Verify against published IP ranges. User agents are trivially forged.

Perplexity and Anthropic both publish address lists; use them before you draw conclusions about who visited.

1. Fetch one important page with JavaScript disabled and read what comes

back. If the answer to the question you want to be cited for is not in that HTML, no crawler policy will save you.

1. Change one thing and wait. Perplexity says up to 24 hours to reflect a

robots.txt change; Google says recrawling can take days to months. Do not evaluate a crawler change on a two-day window.

What to take from this chapter

Access is a prerequisite, not a strategy. Getting it right buys you the possibility of being cited and nothing more - the next chapter is about why being fetched and being cited are separated by a retrieval step that nobody publishes. But getting it wrong is uniquely expensive, because every other thing you do afterwards is measured against a surface the model never saw.

The single highest-value action in this chapter is unglamorous: read your logs, per agent, and find out whether the crawler that puts you in the answer is actually reaching your pages. Most teams have never looked.