Training versus search: the two kinds of AI crawler

Does blocking AI crawlers stop my site being cited?

It depends entirely on which one you block, and the difference is stark. Blocking a training crawler removes your content from a model's training corpus and costs no citations. Blocking a search or user-triggered crawler removes your pages from the retrieval path an assistant uses when it answers, which is where citations come from.

The two chains are separate

The clearest way to hold this is as two pipelines that share a starting point and never meet again.

Training chain
  training crawler fetches page  ->  page enters corpus  ->  model weights change
                                                            (no link, no referral, nothing measurable)

Citation chain
  search crawler fetches page    ->  page enters retrieval index  ->  a question matches it
                                                                  ->  answer names and links it
  OR
  user asks about a URL          ->  user-triggered fetcher reads it live  ->  answer names and links it

Nothing in the training chain produces a citation. Nothing in the citation chain requires the training chain to have run. That independence is the reason the operators issue separate tokens, and it is the reason a single "block AI" rule does damage that its author did not intend.

What the operators actually say

This is not an inference from behaviour. It is written down, in each operator's own words.

OpenAI's pairing is the one most people meet first, and the wording leaves no room: GPTBot crawls content that may be used in training, OAI-SearchBot surfaces websites in ChatGPT's search features.[^openai-split] Anthropic draws the same line, with ClaudeBot collecting content that could contribute to training and Claude-SearchBot analysing content to improve the relevance and accuracy of search responses.[^anthropic-split]

Perplexity is the strongest case, because it makes the negative claim. It documents no training crawler and states that PerplexityBot is not used to crawl content for AI foundation models.[^perplexity-no-training] For a site owner acting on a training objection, blocking PerplexityBot achieves nothing except removing the crawler that would have linked to them.

The Google and Apple variant: same idea, different mechanism

Google and Apple reach the same separation without running two crawlers.

Google-Extended is a permission, not a fetcher. It governs whether crawled content may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI, and Google states directly that it does not impact a site's inclusion in Google Search and is not a ranking signal.[^google-extended-neutral] That sentence is worth reading twice, because the most common reason people hesitate to disallow Google-Extended is a fear of Search consequences that Google has explicitly denied.

Googlebot sits on the other side of the line, and its scope is wide: preferences addressed to it affect Search including Discover and every Search feature, plus Images, Video and News.[^googlebot-breadth] It is the token where a mistake is most expensive.

Apple's arrangement is identical in shape. Applebot powers Spotlight, Siri and Safari, and the data it crawls may also help train Apple foundation models; Applebot-Extended is the opt-out from the second use without stopping the first.[^applebot-dual-use] One crawl, two permissions, one token per permission.

The third category: user-triggered fetches

Search indexing is not the only citation path, and the other one behaves differently enough to deserve its own treatment.

When someone pastes a URL into ChatGPT, or asks a question that makes an assistant go and read a specific page, a user-triggered agent fetches it live. Perplexity describes exactly this: when a user asks a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response.[^perplexity-user-purpose] That is a citation, produced seconds after the question, with no index in between.

These agents have two properties that matter.

They are the least controllable. OpenAI states robots.txt rules may not apply to ChatGPT-User because the fetches follow user actions,[^openai-user-fetch] and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Meta says meta-externalfetcher may bypass robots.txt for user-requested fetches.[^meta-split] Anthropic is the exception and documents Claude-User as honouring robots.txt.

They are the most valuable. A user-triggered fetch happens because a specific person is asking about something your page answers. It is the highest-intent crawl your server receives.

The consequence is uncomfortable but worth stating plainly: for several operators, robots.txt is not the lever that controls whether an assistant reads your page on a user's behalf. If a page must not be readable by an assistant, the control has to be authentication or an origin-level rule, not a directive.

What each choice actually costs

Setting the tradeoff out as a table is more honest than a recommendation, because the right answer depends on a rights position this guide cannot hold for you.

Row two is the accidental configuration. Nobody chooses it; people arrive at it by writing a wildcard rule aimed at training, or by naming only the training tokens they had heard of and leaving the search tokens to a restrictive default.

A worked example: what one wildcard rule does

Consider the shortest robots.txt anyone writes when they decide they object to AI training. Four lines, one intention, and an effect that spans every operator in this guide.

User-agent: *
Disallow: /

Traced against the documented agents, that rule reaches far past training.

Six of the seven outcomes are not what the author wanted, and one of them is a business emergency. The rule that expresses the actual intention names the training tokens individually and leaves the rest alone — which is what the robots.txt chapter provides as a copyable block.

Sorting an unfamiliar agent yourself

New tokens appear. Four questions, asked against the operator's own documentation and nothing else, put almost any of them on the right side of the line.

1. Does the operator's description contain the word "train"? If the stated

purpose is training or model improvement, it is in the training chain and blocking it costs no citations.

1. Does the description mention search, surfacing, indexing or linking? Those

verbs mark the citation chain. Perplexity's "surface and link websites" and OpenAI's "surface websites in search results" are the canonical phrasings.

1. Is the fetch described as following a user action? Then it is a

user-triggered agent: highest intent, and possibly not bound by robots.txt.

1. Does the operator publish an IP range file or a reverse-DNS suffix for it?

An agent with no verification story is one you cannot confirm in your logs, whatever it claims about itself.

If the documentation does not answer question one or two, the honest conclusion is that you do not know what the agent is for — not that it is safe to assume.

Where the honest uncertainty is

Three things this chapter deliberately does not claim.

That allowing a search crawler produces citations. It makes them possible. Retrieval, ranking and the model's decision to cite are all downstream, and no operator publishes how they work.

That blocking a training crawler removes existing training data. Every statement here is about future crawls. What is already in a corpus is a separate question, and Common Crawl's is the case where the indirection is widest: it publishes an open repository for research,[^ccbot-open-corpus] and who uses the corpus afterwards is not something a robots.txt line reaches.

That the categories are permanent. Amazon's framing shows why: Amazonbot is documented as improving products and services, with content that may be used to train Amazon AI models — a single agent doing both jobs.[^amazonbot-dual] Any operator could make that move. The quarterly review of this guide exists to catch it when one does.

The directives that implement each row of the tradeoff table are in the robots.txt chapter, and the tokens they address are in the user-agent reference. The decision itself, with a recommendation for the common cases, closes the guide in what to allow and what to block.

[^openai-split]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^openai-user-fetch]: OpenAI, crawlers documentation, retrieved 2026-09-03. [^anthropic-split]: Anthropic support documentation, retrieved 2026-09-03. [^perplexity-no-training]: Perplexity crawler documentation, retrieved 2026-09-03. [^perplexity-user-purpose]: Perplexity crawler documentation, retrieved 2026-09-03. [^google-extended-neutral]: Google Search Central, common crawlers, retrieved 2026-09-03. [^googlebot-breadth]: Google Search Central, common crawlers, retrieved 2026-09-03. [^applebot-dual-use]: Apple support, About Applebot, retrieved 2026-09-03. [^ccbot-open-corpus]: Common Crawl, CCBot, retrieved 2026-09-03. [^amazonbot-dual]: Amazon developer documentation, Amazonbot, retrieved 2026-09-03. [^meta-split]: Meta for Developers, web crawlers, retrieved 2026-09-03.