llms.txt

llms.txt is a proposed convention for publishing a short Markdown file at /llms.txt that gives an agent a curated, expert-level map of a site: what the project is, and where the detail lives. It was proposed by Jeremy Howard in a document published on 3 September 2024, now in a v2 revision dated 10 August 2026.

The argument behind it is about cost, not permission. Web pages are built for people: an HTML page wraps its information in navigation, ads and JavaScript, and converting that back into clean text is difficult and imprecise. Context windows are larger than they were and still too small for most sites in their entirety, and every wasted token costs time and money. So the proposal asks a site to do the extraction once, deliberately, rather than have every agent do it badly.

What the file looks like

The spec fixes an order, so the file can be read by a parser as well as by a model:

• an optional byte-order mark;
• an H1 with the name of the project or site — the only required section;
• a blockquote with a short summary containing the key information needed to understand the rest of the file;
• zero or more Markdown sections of any type except headings, giving more detail about the project and how to interpret the files it points at;
• zero or more H2-delimited sections, each a list of links in the form [name](url) followed optionally by a colon and notes.

By convention a section headed "Optional" holds secondary links an agent can skip when a shorter context is needed. The file itself is meant to stay small enough to fit in context; the detail lives behind the links and is fetched only when needed.

The second half of the proposal, which is usually ignored

llms.txt is normally discussed as a single file. The proposal actually asks for two things, and the second one does more work: that pages an agent might need also offer a clean Markdown version at the same URL, either with .md appended (page.html.md) or with the extension replaced (page.md).

To make those discoverable it recommends standard link relations — rel="alternate" type="text/markdown" pointing at the Markdown version, and rel="describedby" pointing at the llms.txt file that covers the page. Both can be <link> elements or an HTTP Link: response header, and the header form works for non-HTML resources and can be set in server or CDN configuration without touching any page.

Why not /.well-known/

The proposal considers RFC 8615 well-known URIs and rejects them for a specific reason: well-known URIs exist only at the origin root, and many authors control only a path on a shared host. A project site on a shared static host can publish files in its own directory but can never add one to the host's /.well-known/. Placing the file like index.html — describing the path where it sits — is something a single root location cannot express.

How it relates to the files you already have

It is designed to coexist rather than replace. robots.txt is about access and is consulted before a fetch; llms.txt is consulted on demand by an agent that is already allowed to read you. XML sitemaps enumerate indexable human-readable pages for search engines; the proposal notes they usually will not list Markdown versions, cannot include useful external URLs, and in aggregate are too large for a context window.

The original expectation, which the v2 revision reports as borne out, was that llms.txt would matter for inference rather than training — that is, for the moment an agent is answering a question, not for a training run.

What it is worth to a publisher

Two honest limits are worth holding onto.

First, it is downstream of access. An agent that is disallowed in robots.txt, or blocked at the CDN, never reaches your llms.txt. Fixing discovery before fixing access is the wrong order.

Second, it is a proposal, not a standard, and there is no ranking authority behind it. The measurable benefit is narrower and real: agents that do follow it spend fewer tokens finding the right page, and are pointed at prose you wrote for that purpose rather than at a rendered shell. That matters most where the corpus is documentation, which the v2 revision says is where the files are used most heavily, with coding agents following them to find API references and tutorials.

If you publish one, keep it small, link only to pages that are genuinely useful without their surrounding chrome, and treat it as part of the same job as crawling and indexing hygiene rather than as a separate growth tactic.