ChatGPT optimization
Making a page eligible to be fetched, retrieved and cited by ChatGPT's search features, and the reasons a page that ranks well on Google is still never quoted.
Karl-Gustav Kallasmaa, Founder & CEOLast updated ChatGPT optimization is the work of making a page eligible to be fetched, retrieved and cited by ChatGPT's search features. It is an access-and-retrieval problem before it is a writing problem: a page that OpenAI's search crawler may not fetch cannot be cited however good it is.
The agents, and what each one decides
OpenAI documents three separate agents, and confusing them is the most common and most expensive mistake in this area.
- OAI-SearchBot — "used to surface websites in search results in ChatGPT's search features". OpenAI states plainly that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links". This is the agent that governs whether you are citable — not whether your URL can ever be surfaced.
- GPTBot — "used to crawl content that may be used in training our generative AI foundation models". OpenAI describes disallowing it as indicating that a site's content should not be used in training. This is a licensing and rights decision, not a visibility one.
- ChatGPT-User — the agent behind user-initiated actions such as question answering and GPT Actions. OpenAI notes that "because these actions are initiated by a user, robots.txt rules may not apply".
Each publishes its own user-agent token, including a distinct form used when requesting robots.txt itself. Treating them as one "OpenAI bot" in a robots.txt file is how a site removes itself from ChatGPT search while intending only to opt out of training.
Why a page that ranks well is still never quoted
Because ChatGPT search does not answer from an index of ranked pages; it answers from passages a model has just read. That changes what "optimised" means at three points.
Access. A retrieval step is an HTTP fetch by a named agent. If OAI-SearchBot is disallowed, or the page is behind auth, or the content only exists after client-side rendering, the passage never enters the candidate pool. See robots.txt for how to answer each agent separately, and crawling and indexing for the mechanics.
Selection. What gets selected is a passage, not a page. A section that depends on three paragraphs above it is correct on your site and wrong once lifted out. This is a property of retrieval-augmented generation generally, not of ChatGPT specifically, which is why the same writing changes help across assistants.
Attribution. A model can use your wording and cite something else, or cite you for a claim you qualified. A citation records what was retrieved, not that it was read correctly. Source citation covers what a link in an answer does and does not prove.
What is actually under your control
- Answer each agent deliberately. Decide separately about training and about search, and write robots.txt so that the two decisions are visible as two rules. Then wait: OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust.
- Make sections self-contained. One subject per section, a heading that states the claim, and the qualifier in the same sentence as the number it qualifies.
- Keep identifiers literal. Version numbers, product codes and prices should be text on the page, not implied by context or rendered from a script.
- Make the page cheap to fetch. Server-rendered, unauthenticated, fast. A page that needs JavaScript execution to show its content is a page that may return nothing.
- Sample the answers. The only way to know what is said about you is to ask the questions your buyers ask and read what comes back, with the sources.
Failure modes
- Blocking the wrong token. A blanket
Disallowaimed at training removes you from search answers as well. - Assuming robots.txt covers everything. It does not cover user-initiated fetches, by OpenAI's own statement — so "we blocked the bots" is not a privacy control.
- Testing too early. Checking the same day you changed robots.txt tests the propagation window, not your rule.
- Optimising for a ranking that does not exist. There is no published ChatGPT ranking factor list, and content built around a guessed one is content built around nothing. AI search and GEO describe what can be observed instead.
- Writing pages instead of passages. The most common quality failure: a well-argued article whose key claim only makes sense after the preamble.
Frequently asked questions
Which crawler controls appearance in ChatGPT search?
OAI-SearchBot. Opting it out removes the site from ChatGPT search answers, though OpenAI says it can still appear as a navigational link.
How long does a robots.txt change take?
About 24 hours, per OpenAI's crawler documentation.
Does blocking GPTBot remove me from answers?
No. GPTBot is the training crawler; search eligibility is OAI-SearchBot's.
Why was my disallowed page fetched anyway?
Because user-initiated fetches by ChatGPT-User may not follow robots.txt.
Terms related to ChatGPT optimization
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
How an AI answer attributes what it says to the pages it read, and why a citation is a distinct outcome from a click or a mention.
The architecture that retrieves documents at query time and has a model write from them, and the reason your page can be quoted without ever being trained on.