Robots.txt
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
Karl-Gustav Kallasmaa, Founder & CEOLast updated Robots.txt is the file at the root of a host that tells automated clients which parts of the site they may fetch. It stopped being a convention and became a standard in September 2022, when the IETF published RFC 9309, the Robots Exclusion Protocol. That is a more consequential fact than it sounds, because the RFC settles the behaviours that used to vary between crawlers.
What the standard actually requires
Four rules from RFC 9309 are worth knowing by heart, because each one has produced a real outage somewhere.
- Size. A parser's limit must be at least 500 kibibytes. Content beyond a crawler's limit may simply not be read, which is how a machine-generated file with thousands of rules silently loses its last rules.
- Redirects. Crawlers should follow at least five consecutive redirects, even across authorities. A robots.txt behind a longer chain may never be reached.
- Caching. Crawlers should not use a cached robots.txt for more than 24 hours. A rule change is therefore not instant, and rolling one back is not instant either.
- Failure modes. A 4xx response means unavailable, and a crawler may then access any resource. A 5xx means unreachable, and the crawler must assume complete disallow unless it holds a cache roughly 30 days old. An error page where robots.txt should be is a site-wide block, not a neutral state.
Why it became an AI question
Robots.txt is now where a site answers a question it did not have to answer before: whether a crawler may take content for model training, for search retrieval, or for both. Vendors deliberately split those jobs across separate user agents so the answers can differ.
OpenAI documents GPTBot as used to crawl content that may be used in training its generative AI foundation models, and states that disallowing GPTBot indicates a site's content should not be used in training; OAI-SearchBot is separately described as used to surface websites in search results in ChatGPT's search features. Perplexity documents PerplexityBot as designed to surface and link websites in its search results and explicitly not used to crawl content for AI foundation models.
That separation is the reason a blanket Disallow: / for every AI agent is almost always the wrong instrument: it answers the training question and the discovery question with one word. GPTBot vs OAI-SearchBot, CCBot vs GPTBot and Google-Extended vs Googlebot each work through one of these pairs.
There is also a category robots.txt does not govern. Both vendors state that their user-initiated fetchers may not follow robots.txt rules, because a person asked for that page specifically. See PerplexityBot vs Perplexity-User.
How it differs from noindex
This is the most expensive confusion in the whole area, and the rule is one sentence long.
Robots.txt controls fetching. A robots meta rule controls what is done with what was fetched. Google's robots meta documentation states that if a page is disallowed from crawling through robots.txt, any information about indexing or serving rules will not be found and will therefore be ignored.
So blocking a page in robots.txt does not remove it from an index; it prevents the crawler from ever reading the noindex that would have removed it. To take a page out, allow the crawl and serve noindex. To keep a page but suppress its snippet, allow the crawl and use nosnippet or max-snippet — noting that Google says the max-snippet limit applies to its AI features too. noindex vs nosnippet is the longer version. And robots.txt is not a security control: it is a published list of the paths you would prefer nobody fetched.
How to act on it
- Read your server logs and list the AI user agents actually reaching you before you write a single rule.
- Answer training and retrieval separately, per agent, and record the reason in a comment in the file.
- Keep the file small and hand-reviewed. Nothing important should sit near a 500 kibibyte limit.
- Monitor the response code. A 5xx on robots.txt is a full-site disallow under the standard.
- Remember the 24 hour cache window when you change anything, and do not judge the effect the same afternoon.
- Point to your sitemap from the file so discovery and access are described in one place — see XML sitemaps.
Frequently asked questions
Is robots.txt a real standard?
Yes, RFC 9309, published September 2022.
How large can the file be?
Parsers must handle at least 500 kibibytes; content beyond a crawler's limit may be ignored.
Does blocking a page remove it from search?
No. The crawler cannot then read the noindex rule that would have removed it.
What happens on a 5xx?
The crawler must assume complete disallow, so a server error is a site-wide block.
Terms related to Robots.txt
The two separate stages that decide whether a page can be retrieved at all, and the reason a serving rule on a blocked page is never read.
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.
The corpus a model learns from, how it differs from what an assistant retrieves at query time, and how publishers have actually responded.
A machine-readable list of the URLs you want discovered, bounded by the protocol at 50,000 URLs and 50MB per file, and a hint rather than an instruction.