Perplexity AI
An answer engine that retrieves live web documents and writes cited answers from them, via two documented agents — one bound by robots.txt and one that, per its own docs, generally is not.
Karl-Gustav Kallasmaa, Founder & CEOLast updated Perplexity is an answer engine: it retrieves documents from the live web in response to a question and has a language model write an answer from them, with links to what it fetched. For a publisher, the important detail is not the interface but the two agents behind it, because they follow different rules and one of them is not governed by your robots.txt at all.
The two agents, in Perplexity's own words
Perplexity documents two user agents.
PerplexityBot is "designed to surface and link websites in search results on Perplexity", and the documentation states it "is not used to crawl content for AI foundation models". It identifies itself as PerplexityBot/1.0 and its addresses are published at https://www.perplexity.com/perplexitybot.json.
Perplexity-User "supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response." It identifies as Perplexity-User/1.0, with addresses at https://www.perplexity.com/perplexity-user.json.
The governing difference is stated in the same document: PerplexityBot is the agent Perplexity asks you to allow in robots.txt — "we recommend allowing PerplexityBot in your site's robots.txt file and permitting requests from our published IP ranges" — while Perplexity-User "generally ignores robots.txt rules", on the reasoning that a person's request initiated the fetch. Perplexity also notes that changes to crawler settings may take up to 24 hours to be reflected.
What that means for a publisher
Three consequences, and they are the ones people get wrong.
A robots.txt block is not an opt-out of answers. Disallowing PerplexityBot removes you from the crawled index that supplies unprompted results. It does not stop a user-initiated fetch, because that path is documented as not bound by robots.txt. The two decisions — be indexed, be fetchable on request — are separate and must be made separately. See robots.txt.
Blocking is not neutral. Removing the crawler does not remove the topic. Answers about your category continue to be written, from whatever sources remain reachable — competitors, directories, forums, and older material that quotes you. The realistic outcome of a block is a description of you assembled from documents you do not control, which cannot be corrected by editing your own page.
Verification is by IP, not by string. The user-agent header is self-asserted. Anyone can send PerplexityBot/1.0. If you are counting agent traffic to decide anything, resolve requests against the published address lists first, or your log will contain scrapers wearing the name.
How a page gets cited here
The mechanism is ordinary retrieval-augmented generation: fetch, select passages into a context budget, generate, attach links. That means the selected unit is a passage, not a page, and it means the failure points are ordered. A page that cannot be fetched loses at step one regardless of quality. A page that is fetched but written as one long argument loses at selection, because the chunk that would answer the question also contains three other subjects.
The citation itself records what was retrieved — not that the model read it correctly. A link to your page under a sentence that misstates your pricing is a generation failure sitting on a successful retrieval, and the fix is usually to make the correct sentence self-contained on the page, so the passage carrying the claim also carries its qualifier. See source citation.
Failure modes
- Reading one answer as the state of the world. Answers vary run to run. A single Perplexity response is one observation; a rate over a defined prompt set with a stated sample size is a measurement. See citation probability.
- Measuring immediately after a robots change. Up to 24 hours, per the documentation.
- Assuming "not used for foundation models" means "not stored". The statement is about training use of PerplexityBot's crawl, and says nothing about how index content is retained or summarised.
- Treating the developer APIs and the consumer product as one measurable surface. The platform documents distinct Search, Agent, Router and Embeddings APIs; what an API returns for a query is not automatically what a person sees in the app.
- Optimising copy while the page renders client-side. A retrieval step is an HTTP fetch on a deadline. A page whose content arrives after JavaScript may simply not be there when it is read.
Frequently asked questions
Which agent obeys robots.txt?
PerplexityBot does, and Perplexity recommends allowing it there; Perplexity-User generally ignores robots.txt, per Perplexity's documentation.
Does blocking remove me from answers?
No. It removes indexed crawling, and can leave a short third-party summary in place of your own words.
How do I verify the crawler?
Against the published IP lists at perplexitybot.json and perplexity-user.json, not the user-agent string.
Does a citation mean the page was read correctly?
No. It records what was retrieved. Retrieval and generation fail independently.
Terms related to Perplexity AI
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.
How an AI answer attributes what it says to the pages it read, and why a citation is a distinct outcome from a click or a mention.
The standardised file that tells crawlers what they may fetch, now the main place where AI training and AI retrieval access are decided separately.
The informal name for the chance an AI answer cites a page. No tool computes it; it is only estimated from an observed citation rate over a fixed prompt set.