OAI-SearchBot vs PerplexityBot: the two crawlers that decide answer-engine visibility

The short answer

OAI-SearchBot and PerplexityBot are the retrieval crawlers of the two most prominent answer engines. Neither is its operator's training crawler. Both exist to index your pages so that they can be surfaced, and linked, when somebody asks a question. For the overwhelming majority of sites, both should be allowed.

Where they differ is in what their operators have committed to in public, and those commitments point in different directions.

The same job, two different promises

OpenAI's documentation makes a promise about consequences. It states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. That is a rare thing to find in crawler documentation: an operator writing down, unambiguously, what you lose by refusing it. It converts a robots.txt decision from a judgement call into an arithmetic one.

Perplexity's documentation makes a promise about scope. It describes PerplexityBot as designed to surface and link websites in search results on Perplexity, and it states that the crawler is not used to crawl content for AI foundation models. That is also rare, and also valuable, but it answers a different question. It tells you what the crawler will not do with your content rather than what happens if you refuse it.

Neither promise is better. They are answers to the two objections publishers actually raise — "what do I lose" and "what will you do with it" — and each operator has answered one of them clearly and left the other implicit.

Being precise about that gap matters. A great deal of published advice asserts that blocking PerplexityBot removes you from Perplexity's answers. That is a reasonable inference from the crawler's stated purpose. It is not a citation, because Perplexity has not written it. This page marks that row as unknown rather than filling it with a plausible-sounding claim, and any comparison that fills it without a source is guessing at you.

Why the training separation matters commercially

Both operators have decoupled retrieval from training, and that decoupling is what makes an "allow both" recommendation defensible rather than naive.

At OpenAI the separation is structural: training lives under GPTBot, a different token with a different address list. You can refuse training and accept retrieval, and the two decisions genuinely do not interact. At Perplexity the separation is definitional: PerplexityBot is described as not being used for foundation models at all, so there is no second decision to make about this crawler.

The two mechanisms have different failure modes, and it is worth knowing which risk you are holding. A structural separation survives a policy change badly but visibly — if OpenAI wanted to use OAI-SearchBot data for training, it would have to change what it publishes about a token you already have a rule for. A definitional separation depends on the sentence staying in the documentation. Both are trust, expressed differently; neither is a contract.

Copyable robots.txt directives

Allow both retrieval crawlers explicitly, so that the decision is recorded rather than inherited:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Allow retrieval, refuse training, which is the configuration most companies with a content archive actually want:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Exclude a specific area from both engines without leaving either index:

User-agent: OAI-SearchBot
Disallow: /customers/
Disallow: /internal/

User-agent: PerplexityBot
Disallow: /customers/
Disallow: /internal/

One trap deserves naming because it is the most common way sites lose both engines at once. A wildcard rule for all user agents applies to these crawlers too. If your file opens with a broad disallow and then names exceptions, both of these tokens have to be among the exceptions or you are invisible to both answer engines while remaining perfectly visible to Google — a state that is easy to reach and hard to notice.

Two different indexes, two different reasons to care

A retrieval crawler is only interesting because of what sits behind it, and what sits behind these two is not the same kind of thing.

ChatGPT search is a feature inside a general-purpose assistant that people already use for a great deal that is not search at all. A citation there arrives in the middle of a conversation, often as one source among several supporting a summarised answer, and frequently in response to a question the person would never have typed into a search box — comparative, hedged, full of constraints. "Which of these tools works if my docs are in a git repo and I have no SEO person" is not a keyword. It is the sort of question an assistant answers by assembling sources.

Perplexity is an answer engine by design, and the interface makes the sources structurally prominent. The product's whole proposition is the cited answer, which means the citation is nearer the surface of what the user is looking at.

For a publisher, that difference changes what a citation is worth rather than whether you want one. Being in ChatGPT's sources reaches enormous general usage in a context where attribution competes with a fluent summary. Being in Perplexity's reaches a smaller, more deliberate audience in a context built around the sources. Neither dominates. What both reward is the same underlying thing: a page that answers a real question directly enough to be quoted in isolation.

Being crawled is the floor, not the goal

It is worth being blunt about the limits of a robots.txt decision, because this pair of tokens is often sold as though allowing them were the work.

Allowing OAI-SearchBot and PerplexityBot puts you in the candidate set. It does not put you in the answer. Both engines then choose among everything they have indexed, and that choice is made on whether a passage on your page actually answers the question that was asked, better than the passages on everybody else's.

That has some practical consequences for how you write, and they are the same for both engines. Pages that answer one question well outperform pages that cover a topic broadly, because retrieval operates on passages rather than documents. A heading phrased as the question a person would ask gives the retriever something to match. A first paragraph that answers that heading standalone, without requiring the paragraph above it, is a passage that can survive being lifted out. Tables get extracted cleanly where the same content in comparative prose does not. Dates and named sources travel with the claim and give the model something to attribute.

None of that is a trick, and none of it is specific to AI. It is the same discipline that makes a page useful to a reader in a hurry. The reason it matters more now is that the reader in a hurry might be a retrieval system with no patience for a preamble.

The order of operations, then, is: allow both crawlers, confirm in your logs that both are actually reaching the pages you care about, and only then spend effort on the pages themselves. Skipping the middle step is how teams end up rewriting content that no crawler has visited since the rewrite before.

Verifying that both crawlers reach you

Allowing a crawler is not the same as being crawled. Both operators publish enough to check the difference properly.

Match on the self-identifying URLs — openai.com/searchbot and perplexity.ai/perplexitybot — rather than on version numbers, which change. Then verify the requesting addresses against openai.com/searchbot.json and perplexity.com/perplexitybot.json. This second step is not optional pedantry. User-agent headers are trivially forged, and the traffic that most often gets miscounted as answer-engine crawling is somebody's scraper borrowing a well-known name. Counting it inflates your sense of coverage, which is worse than having no number at all.

What you are looking for in the logs is coverage, not volume. Which of your pages have been fetched by each engine, and how recently. A documentation section that neither crawler has visited in months cannot be quoted in an answer, no matter how good it is, and no amount of allow rules will change that on their own.

Attensira's crawler logs record fetches by agent and URL, which is exactly the per-page coverage view this question needs. The bot access score reads back what your current rules permit, and the robots.txt generator will build a file that says what you meant.

When the honest answer is to block one of them

This page has argued for allowing both, so it should be equally clear about the cases where that is wrong.

If a section of your site contains content that should not be quoted out of context — customer names on a logo wall you have permission to display but not to have summarised, internal-facing documentation that is public only because nobody got round to authenticating it, legal or medical text whose accuracy depends on the surrounding caveats — then excluding that section from both retrieval crawlers is the correct call. Path-scoped rules exist for exactly this, and they cost you nothing elsewhere on the site.

If your revenue is the pageview, the calculation is genuinely different and this page will not pretend otherwise. An answer engine that summarises your article with a link attached is not equivalent to a reader arriving on the page, and no amount of enthusiasm about citations changes the arithmetic for a publisher whose business is advertising impressions. Blocking is a coherent response, and it is a response some serious publishers have chosen after doing the sums.

What is never right is arriving at either outcome by accident — a wildcard rule, an inherited template, or a security review that refused an unfamiliar user agent on principle. Both of these crawlers govern whether people can find you inside the products they are increasingly asking their questions in. That deserves a decision with a name attached to it.

Related comparisons

To understand OAI-SearchBot against OpenAI's own training crawler rather than against a competitor, read GPTBot vs OAI-SearchBot. PerplexityBot's sibling takes the opposite position on robots.txt entirely — see PerplexityBot vs Perplexity-User. The two live fetchers are compared in ChatGPT-User vs Perplexity-User, and Anthropic's arrangement in ClaudeBot vs Claude-User.