ClaudeBot vs GPTBot: two training crawlers, two different bargains

The short answer

ClaudeBot and GPTBot do the same job for the two vendors whose assistants people actually use. Both collect web content that may end up in a foundation model. Both are documented as robots-controlled. Neither one's documentation claims that blocking it will change what the assistant says about you.

That last sentence is the reason this comparison is worth making at all. Almost every conversation about AI crawlers collapses two questions — may this company train on my work and may this company cite my work — into a single robots.txt line. ClaudeBot and GPTBot answer only the first question. The second is answered by other agents entirely, and blocking the training crawler at either vendor leaves that second door exactly as open as you left it.

Same job, different levers

The purposes are near-identical in substance and phrased differently in the documentation. Anthropic describes ClaudeBot as collecting web content that could potentially contribute to training its generative models. OpenAI describes GPTBot as used to make generative AI foundation models more useful and safe. Read charitably, both sentences mean the same thing: this agent gathers material for the model, not for an answer being written right now.

Where they diverge is in what a publisher is allowed to say back.

Anthropic documents ClaudeBot as respecting robots.txt and as supporting crawl-delay. That second directive is unfashionable — it was never part of the original robots exclusion standard and most large operators quietly ignore it — but where an operator commits to honouring it, it converts a binary into a dial. A publisher who does not object to training in principle, and only objects to a crawler hammering a database-backed archive at three in the morning, has something to write.

OpenAI's crawler documentation takes the other route. It gives GPTBot a robots.txt token, a published user-agent string ending in compatible; GPTBot/1.4; +https://openai.com/gptbot, and a machine-readable address file, and it says nothing about crawl-delay. For GPTBot the documented vocabulary is allow or disallow. Anything in between has to be built at your own edge with rate limits, and at that point you are enforcing rather than requesting.

What each operator says blocking actually does

This is the row of the table that people skip and then regret.

Anthropic frames the effect of blocking ClaudeBot in terms of future material: content behind the block is excluded from its AI model training datasets. OpenAI frames disallowing GPTBot as indicating that a site's content should not be used in training generative AI foundation models.

Both are forward-looking statements about data collection. Neither is retroactive, and no honest reading of either sentence supports the common claim that blocking a training crawler removes your content from a model that has already been trained. If a model already knows your documentation, a robots.txt commit today does not unlearn it, and nobody should sell you a tool that says otherwise.

Neither statement mentions answers, citations or visibility. That absence is the finding. At both vendors the visibility consequence lives on a different token: Anthropic documents Claude-SearchBot and Claude-User for search and user-directed fetches, and OpenAI documents OAI-SearchBot as the crawler that surfaces sites in ChatGPT's search features. OpenAI lists four crawlers in total, each with its own token, and the split exists precisely so that these decisions can be made separately.

Telling them apart in logs

GPTBot is the easier of the two to isolate. It announces itself with a stable self-identifying URL in the user agent, and OpenAI publishes a GPTBot-specific address file at openai.com/gptbot.json. Two independent signals, one of which is forgeable and one of which is not, and they are scoped to this one crawler. If you need to prove in an audit that GPTBot specifically stopped fetching after a robots change, you can.

Anthropic publishes crawler addresses too, at claude.com/crawling/bots.json, but as one file covering its crawlers rather than one per agent. The practical consequence is that address verification tells you the request genuinely came from Anthropic, and the user-agent token is the only thing that tells you which Anthropic agent it was. Since user-agent strings are trivially forged, verifying an Anthropic fetch is a two-step where the second step depends on a signal you cannot independently confirm. That is not a criticism of Anthropic's transparency — publishing anything at all puts both vendors far ahead of most crawler operators — but it changes how you write a log filter.

There is a version trap on the OpenAI side worth flagging. GPTBot's published user agent currently carries a /1.4 version number. A firewall rule or log query written against GPTBot/1.0 silently stops matching the day that number changes, and a silent stop looks exactly like a crawler that went away. Match on the token and the self-identifying URL, never on the whole string.

The robots.txt lines

Refuse training at both vendors, keep every retrieval path open. This is the configuration most publishers who care about the training question actually want:

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow: /

Refuse training at Anthropic but only throttle it, using the directive Anthropic documents as supported:

User-agent: ClaudeBot
Crawl-delay: 10

And the configuration nobody writes down but most sites are already running, which is worth committing precisely because it turns a default into a decision:

User-agent: ClaudeBot
Allow: /

User-agent: GPTBot
Allow: /

Two mechanical notes that cause more damage than either vendor's policy ever will. Robots.txt matching is on the token, not on the full user-agent string, so a User-agent: line naming GPTBot is correct and one pasting the whole Mozilla string is not. And robots.txt is per host and per scheme: a rule on your apex domain does nothing for the documentation subdomain, which is usually where your most citable pages live and usually the host whose robots file was generated by a framework default that nobody has looked at since.

Where this decision usually goes wrong

The first failure is symmetry by reflex. Somebody blocks GPTBot because of a headline, then adds ClaudeBot for consistency, then adds the search crawlers of both vendors to be thorough, and ships a commit that refuses training — which was the intention — and also removes the site from two assistants' answers, which was not. The training tokens and the retrieval tokens are separate for a reason. Use the separation.

The second failure is assuming the blocks are enforcement. Both of these documents describe crawlers that honour a published request. That is a good-faith commitment from two operators who have put their names on it, and it is not a firewall. The published address lists exist so you can check whether the commitment is being kept, which is a different and better use of your time than arguing about whether it will be.

The third is treating the decision as permanent. Both pages are living documents: OpenAI's crawler roster has grown, Anthropic's article now describes three agents where it once described one, and a robots.txt written against last year's roster is a robots.txt with holes in it. Re-read both pages when you review this file, and diff them against what you assumed.

What neither token buys you

It is worth being precise about the limits, because overpromising here is how publishers end up angry at a standard that never promised them anything.

Neither directive is a copyright position. A robots.txt line is a machine-readable preference expressed to a well-behaved client; it is not a licence, not a contract, and not a substitute for whatever terms your legal team has written. Treating it as a legal instrument is a category error that has cost people real money.

Neither directive reaches content you do not serve. If your posts are syndicated, mirrored on an aggregator, quoted at length in a forum or republished by a partner, those copies live behind somebody else's robots file and answer to somebody else's decisions. Blocking a training crawler at your origin does nothing about the version of your work that a third party is hosting, and for widely-syndicated publishers that copy is often the more heavily crawled one.

Neither directive controls what a person pastes into a chat window. A user who copies your entire pricing page into an assistant has moved that content across a boundary no crawler policy touches. Both vendors document separate user-triggered agents precisely because that traffic is a different thing with different rules, and Anthropic and OpenAI both say plainly that user-initiated fetches are governed differently from automated crawling.

And neither directive is a growth strategy in either direction. Blocking training does not make you more valuable; allowing it does not make you cited. The path from being crawled to being quoted runs through whether your pages answer a real question better than the pages currently being quoted, and no line in a robots file shortens it.

Reviewing this decision on a schedule

Both of the documents this page is built on are living pages that change without announcement. OpenAI's crawler roster has grown over time and each addition arrives with its own token, which means a robots.txt written to be exhaustive last year is merely partial today. Anthropic's support article now describes three distinct agents where earlier versions described the crawler as a single thing.

The practical habit is small: put a quarterly reminder against your robots file, re-fetch both operator pages, and diff the crawler list against the tokens you have rules for. A new agent from a vendor you have already made a decision about is the easiest thing in this whole area to miss, because nothing breaks and no log line screams. It simply crawls, under a name your file has never heard of, doing a job you already decided you did or did not want.

Confirming the change landed

A robots.txt edit is a statement of intent; your access log is the fact. After changing either token, the only question worth answering is whether the agent you disallowed stopped fetching and the agents you left alone kept fetching — per URL, not in aggregate, because a rule scoped to a path behaves very differently from one scoped to a host.

Attensira's crawler logs record which AI agent fetched which URL and when, so a change to ClaudeBot or GPTBot access can be checked against observed behaviour rather than assumed. To read your current rules before changing them, the robots.txt generator and the bot access score will tell you what your file permits today.

Related comparisons

For the sibling split inside each vendor rather than the rivalry between them, read ClaudeBot vs Claude-User and GPTBot vs OAI-SearchBot. For the two vendors' search crawlers head to head, see Claude-SearchBot vs OAI-SearchBot. And for GPTBot against the open-archive crawler that predates every commercial training bot, see CCBot vs GPTBot.