Anthropic
The AI safety company that builds the Claude models, and the three separately-named web crawlers a publisher has to answer if they want Claude to read their pages.
Karl-Gustav Kallasmaa, Founder & CEOLast updated Anthropic is an AI safety and research company that builds the Claude family of large language models. For anyone trying to be read and cited by an AI assistant, it is also the operator of three distinct web crawlers, and the difference between them is the most consequential thing on this page.
The company's own company page describes it as building frontier AI systems that are reliable, interpretable and steerable, and states that it operates as a Public Benefit Corporation whose charter names the responsible development and maintenance of advanced AI for the long-term benefit of humanity. Governance is split between a conventional board and a Long-Term Benefit Trust, whose trustees have representation in board elections.
The models, and the numbers that constrain a publisher
Anthropic's models overview documents the current lineup as Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5 and Claude Haiku 4.5. The first three carry a 1M-token context window with a 128k-token maximum output; Claude Haiku 4.5 carries 200k tokens with a 64k maximum output. The same table gives reliable knowledge cutoffs of June 2026, May 2026, January 2026 and February 2025 respectively.
Two facts in that paragraph do real work.
The cutoff is why what you publish today cannot reach a model's weights in time to matter. Anything written after a model's training data cutoff can only reach an answer by being fetched at query time — which makes retrieval, not training, the channel a publisher can actually influence. See AI training data.
The window size is why being fetched is still not enough. Anthropic's context windows guide is explicit that more context is not automatically better: as token count grows, accuracy and recall degrade, a phenomenon it names context rot. A million tokens is large and still finite against a corpus that is not, so a selection step always sits between your page and the answer. What competes is the passage, not the document.
The three crawlers, and why a blanket block is a mistake
Anthropic's crawler documentation names three user agents with three separate purposes:
- ClaudeBot — collects web content that could potentially contribute to training the models. Disallowing it signals that future material should be excluded from training datasets.
- Claude-User — fetches a page because a person asked Claude something that required reading it. Disallowing it prevents content retrieval for user-directed web requests.
- Claude-SearchBot — navigates the web to improve search result quality. Disallowing it prevents indexing for search.
These are three decisions, not one. A publisher who wants to stay out of training data but remain quotable in answers blocks ClaudeBot and leaves the other two alone. A rule written against a wildcard user agent, or a WAF rule matching the string Claude, collapses all three into a single opt-out and removes the page from answers as a side effect of a training decision nobody meant to extend that far. This is the most common self-inflicted visibility failure in robots.txt, and it is silent: nothing reports the answers you were excluded from.
Two operational details follow. Anthropic publishes its crawler IP ranges at claude.com/crawling/bots.json, so verification of claimed crawler traffic belongs against that list rather than against a forgeable header. And throttling is expressed with the non-standard Crawl-delay extension, which Anthropic documents and commits to respecting where appropriate — a commitment, not a guarantee derived from any specification.
What this means if you want to be cited
Being cited by Claude is downstream of three separable conditions, and they fail independently:
- Fetchability. Claude-User and Claude-SearchBot must be allowed, and the page must return quickly to an unauthenticated request. A page behind a login or a client-side render is not a candidate regardless of quality.
- Selectability. The retrieved unit is a section. A heading that states its claim, with the qualifier written into the same sentence as the claim, survives being lifted out; a paragraph that depends on the two above it does not.
- Correctness under extraction. The model writes from what it retrieved. A passage that is accurate only in the context of the surrounding article will be reproduced inaccurately, and the citation will point at you.
For how this fits the wider set of assistants and their agents, see AI search.
Frequently asked questions
What is Anthropic?
An AI safety and research company that builds the Claude models and operates as a Public Benefit Corporation.
Do all three Anthropic crawlers do the same job?
No. ClaudeBot collects training data, Claude-User fetches for a person's question, Claude-SearchBot indexes for search — each blockable separately.
Does blocking ClaudeBot remove me from Claude's answers?
No. It signals exclusion from future training datasets. Answer-time retrieval runs under Claude-User and Claude-SearchBot.
How big is the context window?
1M tokens on Claude Fable 5.1, Claude Opus 5 and Claude Sonnet 5; 200k on Claude Haiku 4.5, per Anthropic's models overview.
Terms related to Anthropic
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.
The corpus a model learns from, how it differs from what an assistant retrieves at query time, and how publishers have actually responded.
The token budget a model can reference in one request, what counts against it, and why a bigger window does not remove the need to retrieve selectively.
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
The units a language model actually reads and writes, why they are not words, and why every limit and price you meet is denominated in them.