ClaudeBot vs Claude-User: Anthropic's training crawler and its live fetcher

The short answer

ClaudeBot and Claude-User are both Anthropic agents, and the difference is when they run. ClaudeBot crawls on Anthropic's schedule to collect content for training its models. Claude-User fetches a page in the moment somebody asks Claude a question. Anthropic documents both as honouring robots.txt.

That last sentence is why this pairing is unusually easy to reason about. With some operators, a robots directive on the user-initiated agent is aspirational. Here it is documented as binding, which means your file actually decides the outcome.

Three agents, not two

Anthropic names three: ClaudeBot for training, Claude-User for user-initiated fetches, and Claude-SearchBot, which navigates the web to improve search result quality and whose blocking, Anthropic says, may reduce visibility in search responses.[^anthropic-third-agent] This page compares the first two because they are the pair people conflate, but a robots file that handles only two of the three has left the most citation-relevant one to a default.

The three-way split maps onto three genuinely different questions. Do you want your content in the training corpus for a future model? Do you want a person's live question about you answered from your actual page? Do you want to be indexed for retrieval at all? A publisher can answer no, yes and yes — and the token structure lets them say exactly that.

Where Anthropic's position is unusual

The interesting field on this page is robots.txt compliance for the user-initiated agent. Anthropic documents Claude-User as honouring robots.txt and describes the token as the mechanism by which site owners control which sites can be accessed through user-initiated requests. Perplexity, by contrast, documents its equivalent agent as generally ignoring robots.txt precisely because a user requested the fetch. Those are two coherent and opposite readings of the same situation, and they are not a difference of implementation quality — they are a difference of policy.

The practical consequence is that a Disallow under Claude-User is expected to work, and a publisher who writes one should expect fewer fetches rather than an unchanged log. If you want to test whether an operator's stated policy matches its behaviour, this is the cleanest experiment available, because the claim is specific enough to be falsified.

Anthropic also documents support for Crawl-delay, a non-standard extension that many operators ignore. That matters more than it sounds. Crawl-delay converts a binary decision into a dial: a small site with expensive dynamic pages can slow a crawler down without cutting itself out of the answer entirely, and cutting yourself out of the answer is what a Disallow does.

Copyable robots.txt directives

Refuse training, stay reachable when somebody asks Claude about you. This is the configuration most product and documentation sites want:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

Rate-limit rather than refuse, using the extension Anthropic documents:

User-agent: ClaudeBot
Crawl-delay: 1

Refuse everything Anthropic operates, accepting that Claude cannot read your pages during a conversation about you:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

Anthropic notes that these rules must be applied to each subdomain separately, which is the same per-host property that catches people out with every other operator, and it catches out documentation subdomains most of all.

The case for keeping Claude-User allowed

Claude-User only fires because a person is already asking. That is worth sitting with for a moment. Every other agent on this page is speculative: a crawler visits in case the content is useful later. Claude-User visits because the usefulness has already been established by a human question, and the only remaining variable is whether the answer they receive is drawn from your actual page or from whatever the model happens to remember.

For a company whose commercial problem is being described inaccurately in AI answers — outdated pricing, a feature you shipped last quarter, a competitor comparison written by somebody else — Claude-User is the repair mechanism. Disallowing it does not remove you from the conversation. It removes your ability to correct the conversation.

The counter-argument is real and worth stating fairly. A user-initiated fetcher can be used at volume, it can reach content you intended for humans arriving through a front door, and there are business models where a live fetch substitutes for a visit that would otherwise have been monetised. Publishers whose revenue is impressions on the page itself have a genuine grievance here that a documentation site does not. The right answer depends on how you make money, not on a general principle about bots.

Identifying the two agents in your logs

Anthropic publishes the robots.txt tokens but not the full user-agent strings on its crawler page, so the honest instruction is to match the token substring in the user agent — ClaudeBot or Claude-User — rather than a documented complete string. The comparison table above records this as unknown rather than inventing a string, because a made-up user agent in a log filter is worse than no filter: it produces confident zeroes.

Anthropic adds a warning worth repeating.[^anthropic-ip-blocking] Blocking its crawlers by address may not reliably prevent crawling, because an address block can prevent the crawler from reading the robots.txt file that would have told it to stay away. That is a genuinely counterintuitive failure: the firewall rule you added to be extra safe can defeat the polite mechanism you were relying on.

The general lesson generalises past Anthropic. Robots.txt is a cooperative protocol, and cooperation requires that the other party can read the instructions. Layering an aggressive network block underneath a polite text file produces a system where neither mechanism does what you designed it to do: the crawler cannot see the file, so it has no directive to honour, and it may reasonably treat the absence of a readable robots file the way any crawler does. If you want a network-level block, make it a complete one and stop relying on the text file. If you want the text file to work, leave the path to it open.

Crawl-delay as the middle option nobody uses

Most publishers treat this as a binary — allow or disallow — and then discover that the binary does not fit their actual problem. The complaint is rarely "this content must never be read". It is "this crawl is expensive", or "this crawl arrives in bursts that look like an incident", or "our dynamic pages cost real money to render and something is walking all of them".

Anthropic's documented support for Crawl-delay addresses that complaint directly, and almost nobody reaches for it. A delay slows the crawl without removing you from anything. For a site with expensive server-rendered pages, a small delay under ClaudeBot converts an infrastructure problem into a scheduling problem while leaving your content fully available for training, retrieval and live answers.

It is worth being precise that Crawl-delay is not part of the original robots exclusion standard and is not universally honoured. Anthropic documents support for it; other operators do not, and a delay directive under a token that ignores it is inert rather than harmful. So the honest way to use it is per operator: set it where the operator has said they will read it, and do not assume it is doing anything where they have not.

The same reasoning applies to path scoping, which is the other underused middle option. Disallowing /search/, faceted-navigation paths, calendar views and other combinatorially large sections costs you nothing in citation terms — nobody is quoting your faceted product filter — and removes the great majority of wasted crawl. Doing that under ClaudeBot before reaching for a site-wide disallow usually solves the real problem.

What to do this week

Three steps, in order. First, enumerate every host you serve — apex, documentation, changelog, help centre, status page — and pull each one's robots file. Second, decide the training question once, as a company, and apply the ClaudeBot rule everywhere consistently rather than per-host by accident. Third, leave Claude-User and Claude-SearchBot allowed unless you have a revenue-shaped reason not to, and then check the logs.

That third step is the one people skip. A robots.txt change is a request, and the only proof that the request was honoured is a change in observed fetches. Attensira's crawler logs record which agent fetched which URL and when, per agent, which turns "we blocked ClaudeBot" from an intention into a measurement. The robots.txt generator will assemble the file, and the bot access score will tell you what your current rules permit before you change anything.

Reviewing this decision on a schedule

Crawler policy is not a thing you set once. Operators add agents, rename them, and change what they say about them, and a robots file written eighteen months ago is describing a world that no longer exists. Anthropic's own estate is a good illustration: a publisher who wrote rules when only ClaudeBot was documented has, by default, allowed two agents they never made a decision about.

A quarterly review is enough for most sites, and it is three questions. Has the operator published a new token since you last looked? Has the description of an existing token changed in a way that widens or narrows its scope? Does your log agree with your file? The third question is the one that catches real drift, because a rule can be correct and ineffective at the same time — a framework upgrade that starts serving a generated robots file, a CDN that caches an old one, a new subdomain that never got the rule at all.

Record what you decided and why, next to the file rather than in a ticket that will be closed. The most common failure in this area is not a wrong decision; it is a decision nobody can reconstruct, which then survives by inertia long after the reasoning stopped applying.

Related comparisons

OpenAI splits the same problem into training and search rather than training and user-initiated fetching — see GPTBot vs OAI-SearchBot. For the operator whose user-initiated agent takes the opposite position on robots.txt, read PerplexityBot vs Perplexity-User. The two live fetchers face off directly in ChatGPT-User vs Perplexity-User, and the two retrieval crawlers in OAI-SearchBot vs PerplexityBot.