LLM content optimization

LLM content optimization is the practice of publishing so that a language model can fetch a page, retrieve a passage from it, and repeat that passage without breaking it. It is three separate problems — access, extraction, and correctness under quotation — and they fail independently, which is why work aimed at one of them does nothing for the others.

Stage one: access is per named agent

A model reaches a page through an HTTP request from a named crawler, and each vendor runs several with different jobs. Anthropic's crawler documentation names ClaudeBot, which collects web content that could contribute to training; Claude-User, which fetches when a person's query causes a retrieval; and Claude-SearchBot, which analyses content to improve search result quality. OpenAI's bot documentation names GPTBot for training, OAI-SearchBot for surfacing websites in ChatGPT's search features, ChatGPT-User for user-initiated actions, and OAI-AdsBot for validating advertised landing pages.

Two operational facts follow. First, "block the AI crawlers" is not a single decision: refusing a training crawler and refusing a search crawler have opposite effects on whether you can be cited, and a robots rule aimed at the wrong user-agent string achieves neither. Second, the change is not immediate — OpenAI documents that an opt-out from appearing in ChatGPT search results takes approximately 24 hours to take effect, and Anthropic asks that directives be repeated on every subdomain, alongside the non-standard Crawl-delay extension. See robots.txt.

Access also fails for boring reasons that no directive describes: a page that renders its content client-side, a slow response, an interstitial, or a login. A page that cannot be fetched cheaply is not a candidate at any level of writing quality. See crawling and indexing.

Stage two: the retrieved unit is a passage

Inside a retrieval-augmented system, a page is chunked, the chunks are indexed, and a query selects some of them. The page is never the unit; the section is.

That reframes almost every writing decision:

• One subject per section. A section covering four topics produces a chunk that represents none of them well, so it loses to a narrower one.
• Headings that state the claim, not the topic. "Crawl-delay is non-standard" is retrievable; "Considerations" is not.
• Qualifiers inside the sentence they qualify. A caveat two paragraphs up does not travel with the claim, and the claim is what gets lifted.
• Literal identifiers. Version numbers, product codes and exact user-agent strings should exist as text, because vector similarity is good at paraphrase and poor at exact tokens.
• No dependency on earlier sections. If a passage is only true given what came before it, it becomes false when quoted alone.

The test is mechanical: cut any section out of the page and read it cold. If it is still correct and still complete, it survives retrieval. If not, that is a defect regardless of how the page reads end to end.

Stage three: eligibility and what you can withhold

The eligibility path is ordinary. Google's AI features documentation states there are no additional requirements to appear in AI Overviews or AI Mode and no special optimizations necessary, and that a page must be indexed and eligible to be shown in Search with a snippet to be used as a supporting link. It also names the controls that limit what is shown from a page: nosnippet, data-nosnippet, max-snippet and noindex.

Those controls declare a preference to systems that honour them. They are not enforcement, and they do not exist uniformly across every assistant. The consequence is that "be quoted less" is a weak lever and "be quoted correctly" is a strong one — which returns the work to stage two. See source citation for what a citation does and does not prove, and GEO for the broader discipline.

Failure modes

• Optimising prose for a model's supposed preferences instead of for chunk boundaries. There is no documented style bonus; there is a documented eligibility requirement and an observable extraction unit.
• Blocking every named agent to protect content, then treating absence from answers as a ranking problem.
• Changing robots.txt and checking the same day. The effect is not instant, and one vendor documents roughly a day.
• Assuming one vendor's rules generalise. The agent names, the controls and the timings differ per vendor and are published per vendor.
• Writing a page that is correct only as a whole. This is the most common defect and the least visible, because the page reads well to the person who wrote it.
• Treating a citation as validation. It records what was retrieved, not that the passage supported the claim.

Frequently asked questions

Is this just SEO renamed?

No. It shares the indexing and snippet-eligibility layer and differs in the unit selected: a passage, not a page.

Which crawlers matter?

At minimum the ones each vendor publishes — Anthropic names ClaudeBot, Claude-User and Claude-SearchBot; OpenAI names GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot.

Does blocking training stop citations?

Usually not, because training and retrieval use different agents. Blocking the wrong one is the common error.

Can I stay indexed but not be quoted?

Partly, by declaration: Google documents nosnippet, data-nosnippet, max-snippet and noindex, and a snippet-ineligible page is also ineligible as a supporting link.