Claude
Anthropic's family of large language models and the assistant built on them, and the three separately-documented agents that decide whether your page can reach it.
Karl-Gustav Kallasmaa, Founder & CEOLast updated Claude is Anthropic's family of large language models, and the name of the assistant built on them. Anthropic's own documentation describes it as "a family of state-of-the-art large language models developed by Anthropic" rather than a single system — which is the first thing to get right, because the models in that family differ from each other in ways that change what an answer can contain.
The family, and why the differences matter
Anthropic's models overview lists a current lineup — Claude Fable 5.1, Claude Opus 5, Claude Sonnet 5 and Claude Haiku 4.5 — with legacy models still available alongside them. Three published properties change what a given model can say about you:
- Context window. Anthropic documents 1M tokens for Fable 5.1, Opus 5 and Sonnet 5, and 200K for Haiku 4.5, noting that 1M tokens is roughly 555k words on the current tokenizer. This is the budget any fetched page competes for.
- Knowledge cutoff. Anthropic publishes both a reliable knowledge cutoff and a training data cutoff per model — May 2026 for Opus 5, Jan 2026 for Sonnet 5, for example. Anything published after a model's cutoff is invisible to that model unless it is fetched at answer time.
- Retirement date. Anthropic states a commitment date before which a model will not be retired. Models leave, and the answer a customer got last year may come from a model no longer serving.
The practical reading: a question answered by a smaller, older model and the same question answered by a larger, newer one are not the same measurement, and neither is a fair proxy for the other.
The three agents
Anthropic documents three user agents, each doing a different job:
- `ClaudeBot` — "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." This is the training crawler.
- `Claude-User` — "supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent." This is a fetch a person asked for.
- `Claude-SearchBot` — "navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses."
Each is addressed separately in robots.txt, with Disallow: / to block or Crawl-delay: 1 to throttle. Anthropic also publishes the IP ranges its crawlers use at https://claude.com/crawling/bots.json for verification, while warning that "alternate methods like blocking IP address(es)... may not work correctly" and recommending robots.txt instead.
The distinction people get wrong is the same one that catches them on every AI platform: the training crawler and the answering agents are different agents. A rule written to keep your content out of training does not, by itself, remove you from answers — and a blanket block of everything named Claude does. Decide the two separately. See robots.txt and AI training data.
How a page reaches an answer
When Claude answers with the web, it is doing retrieval-augmented generation: fetch documents, place some of them in the context window, write from what was read. Anthropic frames the same idea in its own glossary as augmenting the model with an external knowledge base passed into the context window at runtime.
Three things follow.
Being fetchable is upstream of everything. If Claude-SearchBot or Claude-User cannot retrieve the page, no amount of quality in it matters. Server-rendered, unauthenticated and fast is the baseline.
A large window is not an unlimited one. A 1M-token window sounds like it removes selection pressure. It does not: a corpus is unbounded and a window is not, so something still chooses which passages enter. Anthropic also warns that recall degrades as a window fills, so filling it is not free.
Recency only arrives by fetch. For anything published after a model's cutoff, the retrieval path is the only path. If your page is blocked to the search agent, your new documentation does not exist to Claude no matter how prominent it is elsewhere.
Failure modes
- A blanket block.
User-agent: *with a broad disallow, or a rule naming every Anthropic agent, removes you from answers as a side effect of a training decision. - Treating "Claude said X" as a fact about Claude. Models are non-deterministic and the family is plural. One answer from one model on one day is one observation.
- Reading the wrong cutoff. A model that has never seen your rebrand will describe the old brand confidently, and that is not a hallucination in the usual sense — it is a correctly recalled stale fact.
- Writing for a page instead of a passage. The unit selected into the window is a section. A claim whose qualifier lives three paragraphs away arrives without its qualifier.
- Assuming the citation validates the sentence. Retrieval and generation fail independently; a correct link can sit under a wrong summary.
Terms related to Claude
The token budget a model can reference in one request, what counts against it, and why a bigger window does not remove the need to retrieve selectively.
The corpus a model learns from, how it differs from what an assistant retrieves at query time, and how publishers have actually responded.
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
The architecture that retrieves documents at query time and has a model write from them, and the reason your page can be quoted without ever being trained on.