Meta-ExternalAgent vs GPTBot: two training crawlers with different scopes

The short answer

Meta-ExternalAgent and GPTBot are both AI crawlers you can refuse in one line of robots.txt, and refusing one does nothing about the other. GPTBot's documented job is training. Meta-ExternalAgent's is training or improving products by indexing content directly — two purposes under a single token.

That scope difference, and the fact that only OpenAI publishes an address list you can verify requests against, are the two things worth knowing before you write either rule.

One token, two jobs

The most consequential line in Meta's crawler documentation is the one describing Meta-ExternalAgent as used for training foundation AI models or improving products by indexing content directly. Read it slowly. Indexing content directly is not training. It is the retrieval-shaped use — the thing OpenAI keeps under OAI-SearchBot and Perplexity keeps under PerplexityBot.

Meta has put both under one control. The practical effect is that a publisher cannot say "no training, yes indexing" to Meta the way they can to OpenAI. The token is all or nothing, and every rule you write about it is a decision about both purposes at once.

Whether that is a problem depends entirely on what you want. If you object to training as a practice, one token is simpler and you lose nothing. If you want distribution but not training — the position most companies with something to sell actually hold — Meta's arrangement gives you no way to express it, and allowing the token by default is conceding the training use without ever having considered it.

What each publisher can verify

The verification gap between these two operators is wide and worth stating without softening.

OpenAI publishes GPTBot's address ranges as a machine-readable list. That means a request claiming to be GPTBot can be checked: right user agent, right address, real crawler. Anything else wearing the name is a scraper.

Meta's web-crawlers documentation publishes user agents but no equivalent list. So a request identifying itself as meta-externalagent/1.1 is taken largely at its word. Since a user-agent header is one line anybody can send, this is a genuine limitation — both for a publisher trying to measure what Meta is doing and for one trying to enforce a block at the edge rather than through robots.txt.

The version numbers are worth a note too. Meta publishes 1.1 and OpenAI publishes 1.4. Neither is stable across time. Any filter, firewall rule or analytics query written against a full versioned string is a maintenance liability that will fail silently rather than loudly. Match on the token substring — meta-externalagent, GPTBot — and, where available, on the published address list.

The sibling agents you must not confuse

Both operators run more than one agent, and the taxonomies are different in a way that catches people out.

OpenAI splits by product surface: GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for live user-initiated fetches, OAI-AdsBot for advertising checks. Four tokens, four purposes, each independently controllable.

Meta splits by relationship. Meta-ExternalAgent is the AI crawler. Meta-ExternalFetcher handles user-initiated agentic requests — an assistant navigating a site to complete a task for somebody. FacebookExternalHit crawls the content of a site somebody shared on one of Meta's apps, which is what produces the preview card.

That last one matters commercially and is the reason to be precise. A blanket rule aimed at "Meta's AI" that accidentally catches facebookexternalhit breaks link previews everywhere your content is shared on Meta's platforms. Titles and images stop rendering, shares look broken, and the cause is three tokens deep in a robots file nobody has read in a year. Blocking meta-externalagent alone does not do this. Getting sloppy with a wildcard does.

Two different theories of what a crawler token is for

Behind the taxonomies is a design disagreement that is worth naming, because it determines how much control a publisher ends up with.

OpenAI's tokens answer the question "what will this content be used for". Training, search, live user fetches, advertising checks — four purposes, four tokens, and a publisher can hold a different position on each. The cost of that design is complexity: four rules to write, four to maintain, and four opportunities to get one wrong.

Meta's tokens answer the question "what is the relationship between us". The AI crawler, the agent acting for a user, the fetcher that renders something a person shared. That is arguably the more intuitive split — it maps onto how a publisher thinks about who is knocking — and it is certainly the shorter file. The cost is granularity, and the place it bites is precisely the one that matters most commercially: training and direct indexing arrive under the same token, so the position most companies actually hold has no way to be expressed.

Neither taxonomy is wrong. But a publisher should notice that the taxonomy is not neutral. It determines which positions are available to you, and a position that cannot be expressed in the available tokens is a position you cannot take, however strongly you hold it. When you allow meta-externalagent because you want the indexing, you are also allowing the training, and there is no line you can write that means otherwise.

What a training crawl actually gets you

The argument for allowing these crawlers is usually made badly, in terms of vague goodwill, so here is the concrete version.

A model that has never encountered your company cannot describe it. When somebody asks an assistant about tools in your category and yours is absent from the answer, the immediate cause is frequently not a ranking problem or a robots problem — it is that nothing the model has seen establishes that you exist and what you do. Training data is one of the two ways that changes; retrieval at answer time is the other, and it works far better when the model already has some notion of the entity it is retrieving about.

For an established brand, that argument is weak. The model already knows who they are, and further training mostly benefits the model. For a company that most people have not heard of, it is the whole game, and refusing training is refusing the mechanism by which anyone might come to know you exist.

That is the asymmetry worth sitting with before copying a large publisher's robots file. The organisations most visibly refusing AI crawlers are the ones with the least to gain from being known and the most to lose from being summarised. If neither of those describes you, their policy is not your policy, and adopting it wholesale means paying a real cost to solve somebody else's problem.

Copyable robots.txt directives

Refuse both AI training crawlers while leaving link previews and search alone:

User-agent: meta-externalagent
Disallow: /

User-agent: GPTBot
Disallow: /

Refuse Meta's AI crawler while explicitly preserving the sharing agent, so the intent is legible to the next person who reads the file:

User-agent: meta-externalagent
Disallow: /

User-agent: facebookexternalhit
Allow: /

Refuse training but keep the retrieval crawler that decides ChatGPT search visibility — the configuration that is available at OpenAI and simply is not available at Meta:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Deciding, honestly

The question underneath all of this is whether being in a foundation model is a cost or a benefit to you, and the answer genuinely differs by business.

If your content is the product — original research, journalism, a paid archive, a body of creative work — training is extraction. You supply the value and receive nothing, and both of these tokens are reasonable things to disallow. Meta's broader scope makes that refusal slightly more expensive in reach and slightly more complete in effect.

If your content is marketing for something else, the calculation inverts. A model that has never seen your documentation cannot describe your product, and a company nobody has heard of is not being protected by being unlearnable. For most small software companies, the training crawlers are not the threat. Being absent from the answer is.

Whichever way you decide, apply it consistently and then check. Rules written at different times against different vendors are how a site ends up refusing one company on principle while allowing three others by inattention. Attensira's crawler logs record which agents fetched which URLs, which is how you find out what your file is actually achieving — particularly for Meta, where no published address list exists to verify against. The bot access score reads back your current permissions, and the robots.txt generator will write a file that names each token deliberately.

Auditing a file that predates one of these tokens

Most robots files in production were written before both of these agents existed, extended once when GPTBot became newsworthy, and never touched again. That history produces a specific and very common shape: a deliberate rule for OpenAI, no rule at all for Meta, and no record of why.

The fix is not to add a Meta rule reflexively. It is to establish which of two positions the company actually holds, because they produce different files. Position one is vendor-specific — you have an objection to a named company, and the correct file names that company and leaves the rest alone. Position two is categorical — you object to foundation-model training as a practice, and the correct file names every operator that publishes a training token, revisited whenever a new one appears.

Almost every file that exists today is an accidental version of position one, arrived at by reading a news cycle rather than by making a decision. That is worth surfacing explicitly, because the people who would have opinions about it — legal, the founder, whoever owns the content — are usually unaware that a policy was set on their behalf by whoever last edited a text file.

Write the conclusion down next to the rules. A comment naming the position and the date costs nothing and turns an inherited file into a maintained one.

Related comparisons

For the open-archive route that neither of these vendor tokens reaches, read CCBot vs GPTBot. For OpenAI's own split between training and retrieval, see GPTBot vs OAI-SearchBot. The usage-token design that Apple and Google chose instead of a separate training crawler is covered in Applebot-Extended vs Google-Extended, and Anthropic's three-token arrangement in ClaudeBot vs Claude-User.