AI search
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.
Karl-Gustav Kallasmaa, Founder & CEOLast updated AI search is search in which a language model composes the answer. The system still retrieves documents, but instead of handing you a ranked list to read, it reads them and writes a response, typically naming a small number of the sources it used. Google's AI Overviews and AI Mode are one form of it; ChatGPT search and Perplexity are another, running their own crawlers against their own indexes.
Why the plumbing matters more than the label
The reason to treat AI search as its own category is not that the interface is conversational. It is that a second, separate access decision now sits between your page and the answer, and it is expressed in named user agents rather than in rankings.
The vendors document this precisely. OpenAI's crawler documentation distinguishes three agents: OAI-SearchBot, used to surface websites in search results in ChatGPT's search features; GPTBot, used to crawl content that may be used in training its generative AI foundation models, where disallowing it indicates a site's content should not be used in training; and ChatGPT-User, for certain user actions in ChatGPT and custom GPTs, where OpenAI notes that because these actions are initiated by a user, robots.txt rules may not apply. Perplexity's bot documentation draws the same line: PerplexityBot is designed to surface and link websites in search results and is explicitly not used to crawl content for AI foundation models, while Perplexity-User supports user actions and, since a user requested the fetch, generally ignores robots.txt rules.
Three practical facts fall out of that, and they are the ones most robots.txt files get wrong.
- Refusing training is not the same decision as refusing search. They are separate agents precisely so they can be answered separately. GPTBot vs OAI-SearchBot and AI training vs AI retrieval work through the consequences of each.
- A blanket disallow removes you from the retrieval pool of products you probably wanted to be in.
- A user-initiated fetch is a different category again, and the published policy for it is that robots.txt may not apply. See PerplexityBot vs Perplexity-User.
How it differs from a search engine, and from an assistant
The three terms are used loosely and mean different things.
- A search engine indexes and ranks. Its product is an ordered list of links; the reading is yours.
- AI search indexes, ranks, then reads on your behalf and writes the answer. Retrieval still happens; the presentation of it does not.
- An assistant may answer entirely from its trained parameters with no retrieval at all. In that mode there is no crawler, no citation and no page fetch, and nothing on your site can influence the answer in the current session.
The middle case is the only one you can act on directly this week, and it is the one worth instrumenting. Retrieval-augmented generation is the architecture underneath it.
What can be measured
On Google there is a public number: in a Pew Research Center analysis of 68,879 searches made by 900 US adults in March 2025, 18% of searches generated an AI summary. For standalone assistants there is no equivalent public panel, which is the honest answer to "how big is it". Anyone quoting a share of assistant traffic should be asked for the method before the number.
Because of that gap, the workable measurement is sampling: choose the questions your buyers actually ask, run them repeatedly, and record which sources are named. That produces a distribution you can watch move, rather than a rank you cannot look up.
How to act on it
- Enumerate the AI user agents hitting your site from your own server logs before changing any rules. Decide from evidence, not from a template robots.txt.
- Answer the search crawlers and the training crawlers separately, and write down why.
- Assume user-initiated fetches will reach you regardless, and make sure the page reads sensibly when it is fetched cold with no session context.
- Keep the pages that answer buying questions fast and unauthenticated. A retrieval step that times out is a citation you never had.
Frequently asked questions
What separates AI search from ordinary search?
Where the answer is assembled: AI search retrieves, then has a model write the answer.
Which crawler does an AI search product use?
A dedicated one, and usually several for different jobs, each documented by the vendor.
Do user-triggered fetches obey robots.txt?
Both OpenAI and Perplexity state that robots.txt may not apply to their user-initiated fetchers.
Does blocking the training crawler remove me from AI search?
Not on products that separate the two agents.
Terms related to AI search
Google's AI-generated summary at the top of a results page, and the snippet controls that decide whether your page can appear inside one.
The two separate stages that decide whether a page can be retrieved at all, and the reason a serving rule on a blocked page is never read.
The architecture that retrieves documents at query time and has a model write from them, and the reason your page can be quoted without ever being trained on.
How an AI answer attributes what it says to the pages it read, and why a citation is a distinct outcome from a click or a mention.