llms.txt vs robots.txt: a proposal and a standard

The short answer

robots.txt is an access-control protocol with a specification. llms.txt is a content-curation proposal without one. They are not alternatives, and choosing between them is a category error: one governs what a crawler is permitted to fetch, the other suggests what an agent might find useful once it is already allowed in.

Status: a standard and a suggestion

robots.txt was formalised as RFC 9309 in September 2022, on the IETF Standards Track. That document uses RFC 2119 keywords, which is why so much of what it says is testable. The rules must be accessible in a file named /robots.txt, all lowercase, at the top-level path of the service. A crawler must match allow and disallow paths against the URI, with the most specific match applying, and access is allowed where nothing matches.

llms.txt describes itself, in its own first sentence, as a proposal to standardise on using an /llms.txt file to provide information to help agents use a website. It was published by Jeremy Howard on 3 September 2024 and updated in August 2026 for a second version. It is a serious and well-argued proposal. It is not a standard, and its own site does not claim to be one.

What each file can actually make happen

robots.txt places obligations on the crawler. The interesting ones are the failure modes. On a 5xx response, a conforming crawler must assume complete disallow. On a 4xx, it may access any resource on the server. Cached copies should not be used for more than 24 hours unless the file is unreachable. Parsers must handle at least 500 kibibytes[^robots-size-limit].

Read those four rules together and you get the operational lesson that matters more than any AI-visibility argument: a server that returns 500 for /robots.txt during an incident is instructing every conforming crawler to stop fetching your entire site. That is a real outage with a delayed and invisible cost.

llms.txt places no obligations on anybody. There is no directive to honour, no defined behaviour on failure, no cache lifetime, no parsing limit, and no user agent to address. It specifies a markdown document whose only required section is an H1 with the name of the project, followed by a blockquote summary and optional H2-delimited lists of links. That is a table of contents, and it is a perfectly reasonable thing to publish. It just is not a control.

The adoption question, answered with a date

Adoption is where this pairing gets emotional, so it is worth being precise about what is actually documented.

In favour of llms.txt: its site states that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer docs, and that Mintlify, GitBook and Wix generate the file automatically[^llmstxt-adoption]. That is real and checkable. Note what it shows, though: AI companies publishing the file for their own documentation is evidence about producers, not about consumers.

Against: Search Engine Journal reported on 2 June 2026 that Google's John Mueller described the file as purely speculative for now, observing that it has existed for years and none of the AI systems use it. That is a secondhand report of a statement from one search engine, not a measurement of the whole ecosystem, and it should be read as such. But it is the most specific public statement either way, and it points in the unflattering direction.

By contrast, the evidence for robots.txt consumption is first-party and unambiguous. OpenAI's own crawler documentation says OAI-SearchBot and GPTBot honour robots.txt, and that robots.txt rules may not apply to ChatGPT-User because those fetches are initiated by a user. Google documents Google-Extended as the token governing whether crawled content is used to train future Gemini models, and states plainly that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.

The asymmetry that decides the argument

llms.txt can only ever be an optimisation. Its best case is that some future agent reads it and finds your documentation faster. Its worst case is a slightly stale markdown file nobody fetches.

robots.txt is load-bearing in both directions. Its best case is that you keep training crawlers out while letting answer crawlers in, or the reverse, according to a policy you actually chose. Its worst case is that a typo or a 5xx removes you from every system that honours the protocol, including the ones whose answers you are trying to appear in.

Asymmetric downside is the whole reason to rank these differently. Spend an hour on robots.txt before you spend ten minutes on llms.txt.

Why the comparison keeps getting made anyway

The two files get compared because they look alike and because the hope behind llms.txt is genuinely reasonable. Both are plain-text conventions at a well-known path. Both are addressed to machines. Both promise a small amount of control over an enormous, opaque process. When a new class of automated reader arrives, reaching for the pattern that worked last time is the obvious move, and llms.txt is exactly that move.

The reason it has not landed the way robots.txt did is worth stating plainly, because it is structural rather than a matter of momentum. robots.txt succeeded because the party being constrained had an incentive to comply: a crawler that ignores exclusion rules gets blocked, rate-limited, and eventually sued. There is a cost to defection. llms.txt asks a consumer to do extra work for a benefit that accrues to the publisher. Nothing punishes an agent for skipping the file, and nothing rewards it for fetching one. Conventions with that shape need either a strong coordinating body or a dominant consumer who insists, and llms.txt has so far had neither.

That is an argument about incentives, not about quality. The proposal is well-specified and the second version tightened it further. If a major assistant vendor announced tomorrow that it reads the file, the calculation would change overnight, and the sites that already publish one would be ahead. That is the honest case for spending twenty minutes on it and no more.

Two mistakes that cost real visibility

Blocking the answer crawler while trying to block the training crawler. The crawlers are different user agents with different published purposes, and OpenAI documents both, so a blanket disallow is a choice to be absent from search answers as well as from training. Anyone who wrote a broad rule during the 2023 wave of AI-crawler blocking should re-read it against the current documentation before concluding that their absence from assistant answers is a content problem.

Treating llms.txt as the AI equivalent of a sitemap. A sitemap is consumed; search engines document reading it and act on it. On the public record, llms.txt is not consumed in the same way. Publishing one and then deprioritising the XML sitemap, internal linking or crawl budget is trading a mechanism that demonstrably works for one that might.

A sane policy for a site that wants to be cited

Decide the training question and the answering question separately, because the crawlers are separate. If you want to appear in ChatGPT's search answers, do not disallow OAI-SearchBot, because OpenAI documents that opting out of it removes you from those answers. If you have a view on model training, GPTBot and Google-Extended are the tokens that express it, and Google states that using Google-Extended costs you nothing in Search.

Keep the file small, keep it valid, and monitor that it returns 200. A monitoring check on /robots.txt is cheap insurance against the 5xx rule.

Then, if you like, publish an llms.txt. Point it at your genuinely useful pages — documentation, reference, structured data-rich material — and regenerate it when your docs change. Treat it as a bet with a small stake, not as a visibility strategy. And do not let it replace an XML sitemap, which search engines demonstrably do consume.

A short checklist you can act on today

Fetch your own /robots.txt over HTTPS and read what it actually returns, rather than what your framework was configured to emit. Confirm the status code is 200, confirm the file is at the top-level path in lowercase, and confirm it is well under the 500 kibibyte parsing floor the specification sets.

List every user agent you disallow and write one sentence next to each saying why. Any rule whose justification you cannot state is a rule to delete. Pay particular attention to blanket wildcards written during a period of anxiety about AI crawling, because those are the rules most likely to be excluding you from assistant answers without anybody having intended it.

Add an uptime check on the robots.txt URL specifically. Given the 5xx rule, a robots.txt that errors is a site-wide crawl stop for conforming crawlers, and it is the sort of outage nobody notices because the pages themselves are still serving fine.

Finally, if you publish an llms.txt, generate it from the same source of truth as your documentation so that it cannot drift. A stale map is worse than no map, and a hand-maintained one will go stale.

What neither file does

Neither file makes your content worth quoting. Both are addressed to machines deciding whether and where to look, not to the model deciding what to say. If your pages are crawlable, fetchable and still absent from AI search answers, the problem is the writing: the pages do not contain a specific, attributable sentence that answers the question being asked. That is a content problem, and the root of your domain is the wrong place to look for the fix.