Crawling and indexing

Crawling and indexing are two stages that get spoken about as one, which is why so much time is lost debugging the wrong one. Crawling is an automated client fetching a document over HTTP. Indexing is the system deciding to store, understand and serve it. Each can succeed while the other fails, and the failures look identical from outside: the page is not there.

The ordering rule that explains most incidents

Serving rules live inside the document. That means the document has to be fetched before any rule in it can be obeyed. Google's robots meta documentation states it directly: if a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving rules will not be found and will therefore be ignored.

So the following pairs are not interchangeable, and mixing them up is the most common self-inflicted indexing bug:

noindex vs nosnippet and robots.txt go further into each instrument.

Crawling is now plural

There used to be effectively one crawl that mattered. There are now several, run by different operators for different purposes, and they are answered separately.

OpenAI documents three agents: GPTBot for content that may be used in training foundation models, OAI-SearchBot for surfacing websites in ChatGPT's search features, and ChatGPT-User for certain user actions, noting that because those actions are initiated by a user, robots.txt rules may not apply. Perplexity documents PerplexityBot for search and Perplexity-User for user-initiated fetches, stating that the latter generally ignores robots.txt because a user requested the fetch.

The consequence for indexing is that "indexed" is no longer a single state. Being in Google's index tells you nothing about being retrievable by an assistant that runs its own crawler over its own index. Google-Extended vs Googlebot and OAI-SearchBot vs PerplexityBot compare specific pairs.

The timing nobody accounts for

RFC 9309, the standard behind robots.txt, sets out behaviour that determines how fast any of this responds:

• a cached robots.txt should not be used for more than 24 hours, so changes are not instant in either direction;
• crawlers should follow at least five consecutive redirects, even across authorities, so a long redirect chain in front of the file can hide it;
• a 5xx response means the file is unreachable and the crawler must assume complete disallow, so an error page in that position blocks the whole site.

A "crawling has stopped" incident is far more often one of these three than a penalty.

How to act on it

1. Diagnose the stage before changing anything. Fetch the URL as the crawler would and read the response headers and the meta rules. If the fetch fails, indexing is not your problem yet.
2. Never use robots.txt to remove a page. Allow the crawl, serve noindex, and remove the disallow only after the rule has been seen.
3. Monitor the robots.txt response code, not just its contents. Availability is part of the policy under the standard.
4. Give the crawlers a current list. A sitemap is the cheapest discovery signal you control; see XML sitemaps.
5. Check eligibility separately for AI surfaces. Google states that a page must be indexed and eligible to be shown with a snippet before it can appear as a supporting link in AI Overviews or AI Mode, so a restrictive snippet rule removes a page that is otherwise perfectly indexed.

Frequently asked questions

What is the difference between crawling and indexing?

Crawling fetches the document; indexing decides to store and serve it. They fail independently.

Why is my noindex being ignored?

Because the page is blocked in robots.txt, so the rule is never fetched.

How fast do robots.txt changes apply?

Up to about 24 hours under RFC 9309's caching guidance.

Does Google indexing make me retrievable by assistants?

No. Assistants that run their own crawlers and indexes make that decision separately.