ChatGPT
OpenAI's conversational assistant, and — because it fetches pages while answering — one of the surfaces where a page you publish today can be quoted today.
Karl-Gustav Kallasmaa, Founder & CEOLast updated ChatGPT is OpenAI's conversational assistant: a product in which a person types a question and a large language model writes the answer, drawing on what it learned in training and, in its search modes, on web pages it fetches while answering.
The second half of that sentence is the half that matters to a publisher. A purely parametric answer is written from weights fixed months earlier, and any URL in it is reconstructed from memory. An answer produced after a fetch is written from a document that exists, was retrieved at that moment, and can be linked. That is the difference between being remembered approximately and being quoted accurately, and it is why ChatGPT is a distribution surface rather than only a product.
The four agents, and why they are not interchangeable
OpenAI documents four separate user agents, each with its own job:
- `OAI-SearchBot` — "used to surface websites in search results in ChatGPT's search features." This is the agent that decides eligibility for citation. OpenAI states that sites which disallow it "will not be shown in ChatGPT search answers, though can still appear as navigational links."
- `GPTBot` — "used to make our generative AI foundation models more useful and safe." This is the training crawler. OpenAI writes that "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."
- `ChatGPT-User` — used for "certain user actions in ChatGPT and Custom GPTs," meaning a fetch a person explicitly asked for. OpenAI notes that "because these actions are initiated by a user, robots.txt rules may not apply."
- `OAI-AdsBot` — visits only pages submitted as ads, to validate their safety and relevance.
The practical consequence is that "block ChatGPT" is not one decision. Blocking GPTBot withholds your content from training while leaving you eligible to be cited; blocking OAI-SearchBot removes you from the answers themselves. Sites that intended the first and wrote the second have removed themselves from the surface they were trying to appear on. The rules are independent, and they should be set independently. See robots.txt for the syntax.
Two timing details follow from the same page. Changes to robots.txt take approximately 24 hours to process, so a rule you edit this morning is not testable this afternoon. And because a ChatGPT-User fetch may ignore robots.txt, robots.txt is a statement of preference to crawlers, not an access control — anything that must not be read by a machine needs authentication, not a disallow line.
How a page becomes part of an answer
The mechanism, when ChatGPT is searching, is retrieval-augmented generation: the system issues a query, fetches candidate documents, selects some of them into the model's context, and the model writes from what it just read. Each stage is a filter your page has to survive.
Fetch. An agent has to be allowed and the page has to return quickly. A page that is disallowed, gated, or rendered entirely client-side may never become a candidate, regardless of how good the writing is. See crawling and indexing.
Selection. Only some fetched material fits the prompt. OpenAI documents a 1,050,000-token context window on its current API models — large, and still finite against the open web, so something always has to choose. The unit being chosen is a passage, not a site.
Generation. The model writes a sentence. It may compress two claims into one, drop a qualifier that lived in a different paragraph, or state a limitation as a feature.
Failure modes worth naming
- Wrong split between training and search. The most common self-inflicted wound: a robots.txt rule aimed at training that also removed the site from citation. Read the agent names before writing the rule.
- A qualifier that did not travel. "Free for up to five users" becomes "free" when the caveat lived two paragraphs above the claim. The fix is textual: put the condition in the same sentence as the thing it conditions.
- Confident summary of a stale page. A fetched page is current by definition, so an outdated statement on it is quoted as current. Old pages you have stopped maintaining are still speaking on your behalf.
- Assuming a link is a verification. A citation records what was retrieved, not that the sentence built from it is correct. The two fail separately, and only one of them is something you can edit.
- Measuring the wrong thing. ChatGPT is non-deterministic: the same prompt asked twice can produce different answers. A single observation is an anecdote, not a rate.
What to do about it
Set the crawler rules deliberately, one agent at a time, and wait a day before checking. Write sections that survive being lifted out of the page — one subject each, a heading that states the claim, and qualifiers written inline. Keep identifiers such as version numbers and product names as literal text. And treat what ChatGPT says about you as something to sample repeatedly rather than check once, because the answer that convinced you is one draw from a distribution. AI search covers the wider set of surfaces this applies to.
Terms related to ChatGPT
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
How an AI answer attributes what it says to the pages it read, and why a citation is a distinct outcome from a click or a mention.
The architecture that retrieves documents at query time and has a model write from them, and the reason your page can be quoted without ever being trained on.