Google-Extended vs Googlebot: a control token and a crawler

The short answer

Googlebot is a crawler; Google-Extended is not. Google-Extended sends no requests and has no user-agent string of its own. It is a robots.txt product token that governs whether content Google already crawled may train future Gemini models or be used for grounding in Gemini Apps and Vertex AI.

Google also states, in the same documentation, that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. That sentence is the reason this page exists.

A token is not a bot

Most robots.txt entries name something that will knock on your door. Google-Extended does not. Google documents it as having no separate HTTP request user-agent string, with crawling conducted by existing Google user agents. What the token does is attach a usage policy to those existing fetches.

This has a practical consequence people find genuinely surprising: you will never see Google-Extended in your logs. Not once, not ever, no matter what you write in robots.txt. Teams occasionally add the rule, wait a week, see zero Google-Extended requests and conclude the directive is being ignored. Nothing is being ignored. There was never going to be a request to count.[^google-crawler-categories]

The mental model that works is a pipeline with two valves. Googlebot is the intake valve: it decides whether the bytes are collected at all. Google-Extended is a downstream valve on one specific use of bytes that have already been collected. Closing the downstream valve changes what happens to the water; it does not stop the water arriving.

The asymmetry in what blocking costs

Disallowing Googlebot is one of the most consequential lines you can put in a robots file. Google documents Googlebot as affecting Search, Discover, Google Images, Video and News together — one token, most of your organic channel. Almost nobody objecting to generative training actually wants that outcome, and yet it is what a hastily written blanket rule produces.

Disallowing Google-Extended costs, by Google's own published statement, nothing in Search. That is a remarkably clean position. Operators are usually vague about the consequences of their opt-out tokens; Google has written down that this one is not a ranking signal and does not affect inclusion. You can hold the training objection and act on it without trading away Search traffic, and you do not have to take that on faith from a blog post.

What Google-Extended does cost is reach into Gemini. The token covers training for future Gemini models, training via the Vertex AI API for Gemini, and grounding in Gemini Apps and Vertex AI. Grounding is the part worth pausing on: it is the retrieval-shaped use, the equivalent of what OAI-SearchBot governs at OpenAI. Bundling training and grounding under one token means Google gives you a coarser instrument than OpenAI does. You cannot say "no training, yes grounding" with Google-Extended the way you can with OpenAI's two separate tokens. It is one switch covering both.

Copyable robots.txt directives

Refuse Gemini training and grounding, keep Google Search exactly as it is:

User-agent: Google-Extended
Disallow: /

Refuse the training use on a specific section only — often the right shape for a site with a valuable archive and a promotional front end:

User-agent: Google-Extended
Disallow: /research/
Disallow: /archive/

What you almost certainly do not want, shown here so it can be recognised in a code review rather than copied:

User-agent: Googlebot
Disallow: /

That last block removes the site from Google Search, Discover, Images, Video and News. It is a legitimate choice for perhaps a handful of sites on the internet and an accident everywhere else.

Why the confusion is so common

Three reasons, and they compound.

The first is naming. Google-Extended looks like a bot name. It sits in a list of bot names, in documentation about crawlers, formatted identically to Googlebot and GoogleOther. Nothing about the label signals that it is a different kind of thing.

The second is that every other operator's opt-out is a real crawler. GPTBot fetches. ClaudeBot fetches. CCBot fetches. A publisher who has already written rules for those tokens reasonably assumes Google-Extended works the same way, and then cannot find the traffic.

The third is that Google's crawler estate is genuinely large — Googlebot and its image, video and news variants, GoogleOther and its variants, Google-InspectionTool, Storebot-Google, Google-CloudVertexBot, and Google-Extended sitting among them. Sorting one product token out of that list requires reading carefully at exactly the moment somebody is trying to close a ticket.

Google's crawler estate is larger than the two names in this comparison

Reasoning about Google-Extended in isolation tends to produce a rule that misses the point, because Google runs a lot of clients and several of them touch generative products.

Google's own documentation groups its clients into three categories: common crawlers, which are used for Google's products and always respect robots.txt for automatic crawls; special-case crawlers, which operate under an agreement between the site and a specific product; and user-triggered fetchers, which are part of tools where an end user starts the fetch.[^google-crawler-categories] Google-Extended is not really any of these — it is a product token bolted onto the first category's output.

Two other names deserve attention while you are in the file. GoogleOther is described as a generic crawler used by various internal Google teams with no specific product impact, which makes it the entry publishers most often disallow on the theory that anything unexplained should be refused. Google-CloudVertexBot is documented as supporting site owners building Vertex AI agents, which means a rule against it may block something your own team is trying to build.

The general point is that the AI question at Google is not one token. It is a token plus a set of crawlers, and the answers interact. Deciding Google-Extended without knowing what your file already says about Googlebot, GoogleOther and the inspection tool is deciding one variable in an equation you have not written down.

The limits of what an opt-out achieves

Two honest caveats, both of which apply to Google-Extended and to every equivalent token elsewhere.

It is forward-looking only. The token governs what may be done with content going forward. It does not reach into a model that has already been trained, and no robots directive from any operator does. If your concern is content already absorbed, this is not the mechanism, and anybody telling you otherwise is selling something.

And it is a preference, not a licence. A robots token is a machine-readable signal that an operator has published a policy about honouring. It is not a contract, it creates no obligation you could enforce, and it says nothing about parties who never read your file. Third-party copies of your content sit behind their own robots files, and a syndication partner's permissive rules are indistinguishable from your own from the model's point of view.

None of that makes the token useless. It makes it exactly what it is: a cheap, clearly documented way to record a position with a named operator, whose stated cost to your Search visibility is zero. That is a good deal on its own terms. It is just not the same thing as control.

What to check on your own site

Start by reading the robots file you actually serve, not the one in your repository — a CDN, a framework default or a security product can rewrite it. Confirm that Googlebot is allowed on everything you want in Search, and that nothing about your generative-AI position has leaked into a Googlebot rule.

Then decide the Gemini question deliberately. If your content is your product — original research, journalism, a paid archive — Google-Extended is a cheap and well-documented way to say no, and Google has told you in writing what it will not cost. If your content is marketing for something else you sell, the calculus usually runs the other way: grounding is distribution, and you want it.

Finally, verify what you can and be honest about what you cannot. Googlebot's behaviour is observable in logs; Google-Extended's is not. Attensira's crawler logs will show you Googlebot's fetches alongside every other AI agent's, per URL, which is the half of this picture that can be measured. The robots.txt generator assembles the file, and the bot access score reads back what your current rules permit — useful before you edit, not just after.

A note on how this reads to a reviewer

If you work somewhere with a security or legal function, expect this pair of tokens to come up in a review, and expect the review to reach for the wrong one. The instinct in a risk conversation is to refuse the thing with the scariest description, and Google-Extended's description mentions training a large model on your content, which sounds alarming in a way that "the crawler that puts you in search results" does not.

The useful thing to bring to that conversation is Google's own sentence: Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. It is short, it is the operator's own words, and it settles the only question the business actually cares about, which is whether saying no costs traffic. With that established, the remaining decision is a values decision about generative training, and it can be made on its merits instead of under a vague fear of breaking Search.

The corollary is that the same conversation must never be allowed to slide into a Googlebot rule. Those two token names differ by one word and one hyphen, and one of them is a routine policy preference while the other is a decision to leave Google Search. Put the diff in front of a second person before it merges.

Related comparisons

Apple ships the same design in a slightly different shape, and comparing the two opt-out tokens directly is instructive: see Applebot-Extended vs Google-Extended. For an operator that splits training from retrieval into two independent tokens instead of bundling them, read GPTBot vs OAI-SearchBot. Anthropic's three-token arrangement is covered in ClaudeBot vs Claude-User, and the open-corpus alternative to all of them in CCBot vs GPTBot.