OpenAI

OpenAI is the company that builds the GPT model family and ChatGPT, and — from a publisher's point of view — the operator of four separately named web crawlers whose access rules decide whether a page can be retrieved, trained on, or shown as a source in a ChatGPT answer.

Most glossary definitions of OpenAI describe the research organisation. That framing is not actionable. If you are trying to be cited, the operationally useful description is narrower: OpenAI is a party that fetches your pages under documented user agents, publishes the IP ranges those fetches come from, and honours (or explicitly does not honour) your robots.txt depending on which agent is asking.

The four agents, and why they are not interchangeable

OpenAI's crawler documentation names four, each with a distinct job and a distinct user-agent token:

The distinction that costs sites the most is the first two. GPTBot is the training crawler. OAI-SearchBot is the retrieval crawler. They are separate directives, and a site that blanket-blocks anything OpenAI-shaped in order to opt out of training also removes itself from the pool of pages ChatGPT's search features can surface. That is a legitimate choice, but it is frequently made by accident, in a single line, by someone who intended only the first thing.

ChatGPT-User behaves differently again. OpenAI states that robots.txt rules may not apply to it, because the request originates from a user action rather than from automatic crawling. Any access model that assumes a single OpenAI policy enforced in one place is wrong about this agent.

Each agent also publishes its own IP range file — openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json, openai.com/adsbot.json — which is the only reliable way to tell a real fetch from a spoofed user-agent string in your logs. User-agent strings are trivially forged; the address ranges are the verification step.

Why the knowledge cutoff is the number that matters

OpenAI documents a context window of 1.05M tokens across its current flagship models, and a knowledge cutoff of 30 April 2026 for GPT-6 Astra, with 16 February 2026 for the GPT-5.6 models.

The cutoff, not the window, is the line a publisher should care about. Everything you published before it may exist somewhere in the model's weights, unattributed and unlinkable — see AI training data. Everything you published after it can only enter an answer by being fetched at query time — which means it passes through OAI-SearchBot, which means your robots.txt is the gate. New content and access policy are the same problem.

The window is worth understanding for the opposite reason: it is large enough that people assume selection has stopped mattering. It has not. 1.05M tokens is a per-request budget, not an index. Something still chooses which handful of documents fill it, and that chooser is a retrieval step. See context window for why a bigger budget does not remove the selection stage.

Failure modes

• One line meant to stop training also stops retrieval. User-agent: GPTBot / Disallow: / does not remove you from ChatGPT search. A wildcard Disallow does. Write the agents out by name. See robots.txt.
• Testing a change too early. OpenAI documents up to roughly 24 hours for a robots.txt update to propagate. A block that "did not work" an hour later has not been tested yet.
• Trusting the user-agent string. Anything can claim to be GPTBot. Verify against the published IP range file before drawing a conclusion from your access logs.
• Assuming one policy covers all four. ChatGPT-User is documented as potentially outside robots.txt control. If your intent is to keep content out of ChatGPT entirely, robots.txt alone does not express that intent.
• Optimising the page and not the fetch. A page that requires JavaScript to render its main content, or sits behind an interstitial or a login, may be fetched successfully and still yield nothing quotable.

What this implies for how you publish

Decide the two policies separately and write them separately: whether your content may be used for training, and whether it may be retrieved to answer a question. They are different trades — the first gives away material with no attribution path, the second is the only route by which a page published today can appear in an answer today. Most publishers who think it through want to refuse the first and permit the second, and the robots.txt file is where that distinction either exists or quietly does not.

Then check your logs against the published IP ranges, so you know which of the four is actually reaching you and which is being turned away.