meta-externalfetcher vs Perplexity-User: the fetchers that skip robots.txt

The short answer

Almost every discussion of AI crawlers assumes robots.txt is the control surface. These two agents are the counterexample, and their operators are the ones who told us.

Meta writes that meta-externalfetcher may bypass robots.txt rules, since user-initiated requests drive its functionality. Perplexity writes that Perplexity-User generally ignores robots.txt rules, because the fetch originates from a user request. Neither statement is a rumour, an inference from logs, or an accusation from a rival. Both are published positions on the operators' own documentation pages.

If you want either of these agents stopped, robots.txt is not the tool. That is the entire practical content of this comparison, and everything else follows from it.

The reasoning behind the exemption

The argument both operators are making is worth stating fairly, because it is not obviously wrong.

Robots.txt governs automated crawling. It was designed for agents that traverse a site systematically, and its purpose was to let publishers protect infrastructure from indiscriminate sweeps. A fetch that happens because a specific person pasted a specific link and asked a question is a different kind of event. It is one page, once, on behalf of a human who could have opened the same URL in a browser — where robots.txt would never have applied.

Under that reading, honouring robots.txt for a user-triggered fetch would mean an assistant refusing to read a page its user can read perfectly well unaided, which mostly punishes the user for their choice of reading tool. Browsers do not consult robots.txt. Neither do feed readers or link previews.

The counter-argument is equally real. The fetch is made by a company's infrastructure, at scale, across many users, and the content ends up inside a commercial product. "A user asked" can cover an enormous amount of automated traffic, and there is no published mechanism by which a publisher can verify that any individual fetch was genuinely person-initiated. That is why some publishers regard the exemption as a loophole rather than a principle.

Both positions are defensible and neither is going to be settled by a robots file. What matters operationally is that the exemption exists, is documented, and shapes what your controls can achieve.

Where the two agents genuinely differ

The exemption is shared. The job is not.

Perplexity-User has a narrow documented purpose: a user asks a question, and the agent might visit a web page to help provide an accurate answer. The output of that product is an answer with citations, and Perplexity's sibling crawler is explicitly described as surfacing and linking websites. A Perplexity-User fetch is, in the ordinary case, a reader arriving by an unusual route.

meta-externalfetcher is scoped more broadly. Meta describes it as fetching individual links at a user's request and supporting product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users. That last clause is the one to sit with. Navigating a site to complete a task is a different activity from reading a page to summarise it, and it is the direction the whole industry is moving: agents that fill forms, click through flows and act on a site rather than merely reading it.

If you operate a site with state — a checkout, a booking flow, a signup, an account area — the agentic case is worth thinking about separately from the citation case, and it is not a robots.txt question at all.

What enforcement actually looks like

Since the request layer is out, the options are the ones you already own.

Perplexity publishes an address list for Perplexity-User at perplexity.com/perplexity-user.json, which is the single most useful thing an operator running a robots-exempt agent can do. It means a publisher who genuinely needs this traffic stopped can write a precise rule at the CDN or WAF and know it targets the right infrastructure. It also means the traffic is auditable: you can tell a genuine Perplexity-User fetch from something merely claiming to be one.

Meta publishes no IP-based verification guidance for its agents. So a rule against meta-externalfetcher matches on a user-agent string that anyone can send, which cuts both ways — you will block impostors along with the real thing, and you have no way to confirm which you blocked, or to prove the real agent respected the block.

Before writing either rule, be clear about what you are choosing. Blocking a user-triggered fetcher refuses a request that a person made. They wanted your page and asked a tool to get it. The right call for a licensed-data business or a paywalled archive; a strange call for a company whose problem is that not enough people have heard of it.

Reading the vendor's whole roster, not one line

The most common mistake this pair creates is a false sense of completion. A team decides Perplexity should not have access, adds User-agent: PerplexityBot / Disallow: /, and files the task as done. Perplexity-User is not covered by that line, has never been covered by it, and Perplexity says so on the same documentation page.

Meta produces the same trap with an extra agent. Blocking meta-externalagent and meta-webindexer stops the systematic crawling; meta-externalfetcher continues, because Meta documented it as exempt. Two of three agents honour the file and the third does not.

The habit that avoids this is to work from the operator's roster rather than from a token you remember. Open the vendor's crawler page, list every agent, and write down what your file says about each — including the agents your file says nothing about, which is where new tokens hide. Both of these pages have grown as agents were added, and neither change was announced anywhere your monitoring would have caught.

Load is not the argument

One thing worth setting aside: these agents are not a bandwidth problem. Both are documented as fetching individual links in response to something a person did, not as sweeping a site. A systematic crawler can request thousands of URLs in an afternoon; a user-triggered fetcher requests the one page somebody asked about.

That matters because the load argument is the one people reach for when the policy argument feels awkward to make out loud. If your objection is genuinely capacity, these two agents are near the bottom of your list and a rate limit at the edge handles them without any policy discussion at all. If your objection is that a commercial assistant should not read your content on demand, say that instead — it is a coherent position, and it needs enforcement rather than a request.

The disclosure itself is worth something

It is easy to read both of these entries as bad news and stop there. Worth resisting.

An operator that quietly ignores robots.txt and an operator that publishes a sentence saying it may not apply are in very different positions. The first leaves publishers to discover the behaviour from access logs, argue about it in public, and never quite establish what is happening. The second has handed you an accurate model of the system, which is the precondition for making any decision at all.

Both Meta and Perplexity chose the second. Neither had to. The disclosure invites exactly the criticism this page is making, and both companies wrote it into documentation aimed at the people most likely to object. That is a form of good faith, and it deserves to be read as one — especially since a great deal of user-triggered fetching across the web happens under generic browser user agents that identify nothing at all.

The right response to a documented exemption is therefore not outrage. It is planning: know which agents are exempt, decide whether you care, and if you do, put the control where it will actually work.

A decision procedure that fits on one screen

Start with what the content is. If your pages exist to be found — marketing, documentation, changelog, developer reference — allow both agents, spend zero further time on this, and go and work on whether the pages answer real questions. A user-triggered fetch on that kind of site is a reader.

If some of your content is the product — licensed datasets, subscriber-only research, a members' archive — separate the hosts or the paths first. The fetchers cannot take what you do not serve to anonymous requests, and an authentication boundary is a far stronger control than any crawler directive, robots-respecting or not.

If you have concluded that a specific agent must be refused, write the rule at the edge, use the published address list where one exists, and set a reminder to re-check the vendor's crawler page in a quarter. Both of these rosters have changed before and will change again, and a rule pinned to today's agent list ages into a rule that covers most of the traffic and none of the new traffic.

What no control reaches

Neither an edge rule nor a robots directive reaches a person who copies your page into a chat window; that content has crossed a boundary no crawler policy touches. Neither reaches copies hosted by a third party under someone else's rules. Neither is retroactive.

And neither is a licence. A robots directive is a machine-readable preference made to well-behaved clients, and these two agents are documented as clients that will not treat it as binding. An edge rule is enforcement, which is a stronger thing, and still not a legal position. Whatever your terms say about automated access, they say it independently of both.

Seeing the traffic at all

The awkward property of a robots-exempt agent is that it is invisible to the tooling most companies use to think about crawlers. Robots.txt checkers report on directives. Compliance dashboards report on whether directives were honoured. An agent that is documented as not subject to the file appears in neither, and the only place it shows up is your access log.

Attensira's crawler logs record which AI agent fetched which URL and when, which is how user-triggered fetches become visible as a category rather than a rumour. To see what your current rules cover — and, more usefully, which agents they do not mention at all — the robots.txt generator and the bot access score will read your file back to you.

Related comparisons

For Perplexity's own split between the crawler that honours robots.txt and the fetcher that does not, read PerplexityBot vs Perplexity-User. For Perplexity-User against OpenAI's equivalent, see ChatGPT-User vs Perplexity-User. For Meta's two robots-respecting agents, see meta-webindexer vs meta-externalagent. And for the two answer-engine crawlers that do honour the file, see DuckAssistBot vs PerplexityBot.