Applebot-Extended vs Google-Extended: two opt-outs that crawl nothing

The short answer

Applebot-Extended and Google-Extended are the same idea implemented twice: a robots.txt token that fetches nothing and exists only to say what may be done with content the operator's real crawler already collected. Apple states that Applebot-Extended does not crawl webpages. Google states that Google-Extended has no separate HTTP request user-agent string.

Both companies also publish an assurance that using the opt-out does not cost you search visibility, which is the reason these two tokens are the cheapest AI-training refusals available.

Why a token that crawls nothing is a good design

It is worth appreciating what this pattern solves, because the alternative is worse in both directions.

If an operator ships a separate training crawler, publishers end up with two crawls where one would do, and a publisher who wants to refuse training must accept a second bot on their infrastructure in order to be able to say no to it. If an operator ships nothing, publishers have no way to distinguish "index me" from "train on me" at all, and the only lever is the nuclear one — disallow the search crawler and leave the index.

A usage token avoids both. One crawl, one fetch, and a separate published statement about what the operator will do with what it took. Apple and Google both landed here independently, and it is a better arrangement for a site's bandwidth bill than the vendor-training-crawler model that OpenAI, Anthropic and Common Crawl use.

The cost is verification, and it is a real cost. Every other agent on this site can be checked: you look in the log, you see whether it stopped. These two can never be checked that way, because there was never a request to see. What you are relying on is entirely the operator's published policy. That is not a criticism of either company; it is simply the honest limit of what a publisher can know, and it should be said plainly rather than papered over.

Where the two tokens genuinely differ

The scope difference is the substance of this comparison.

Apple documents Applebot-Extended as determining whether content trains Apple's foundation models for generative features. That is a training control, described as a training control, and the documentation does not describe a retrieval or grounding use.

Google documents Google-Extended more broadly: training for future Gemini models, training via the Vertex AI API for Gemini, and grounding in Gemini Apps and Vertex AI. Grounding is the retrieval-shaped use — the thing that happens when a model fetches or consults your content in order to answer a live question. Bundling it with training means a Google-Extended disallow reaches further than an Apple-Extended one, and further than OpenAI's GPTBot rule, which is a training control with retrieval kept under a separate token.

That has a direct practical consequence for anyone who wants to be cited. With OpenAI you can refuse training and keep retrieval. With Google-Extended you cannot separate them: one token, both uses. If appearing in Gemini answers matters to you, that is an argument for leaving the token alone even if you have refused training elsewhere — and it means "we opted out of AI training everywhere" is a policy that quietly costs more at Google than at OpenAI.

Apple's narrower scope is not automatically better. It is simply narrower, and because the documentation does not describe a grounding use, this page records that row as unknown rather than asserting that Apple's generative retrieval is unaffected. Nobody should tell you what an operator has not written down.

Copyable robots.txt directives

Refuse AI training at both companies while keeping every search surface intact:

User-agent: Applebot-Extended
Disallow: /

User-agent: Google-Extended
Disallow: /

Refuse for a valuable archive only, leaving the rest of the site available:

User-agent: Applebot-Extended
Disallow: /research/

User-agent: Google-Extended
Disallow: /research/

What must not happen, shown so it can be caught in review — the parent crawlers are what carry your search visibility, and they are a completely different decision:

User-agent: Applebot
Disallow: /

User-agent: Googlebot
Disallow: /

Those two lines remove you from Spotlight, Siri and Safari search on one side and from Google Search, Discover, Images, Video and News on the other. The similarity of the token names to the extended ones is exactly why this mistake happens, and why a second pair of eyes on a robots.txt diff is worth more here than almost anywhere else in a codebase.

Why Apple's version is the one people forget

Google-Extended gets written about constantly. Applebot-Extended does not, and the gap has very little to do with the merits of either.

Part of it is where the surfaces are. Google's generative products live in a browser tab next to a search box that publishers have been optimising for two decades, so a Google token lands in an existing workflow with an existing owner. Apple's live inside an operating system — Spotlight, Siri, Safari's own features — where there has never been a webmaster console, a rank tracker, or a job title responsible for how you appear.

Part of it is that Apple's crawler is old news. Applebot has been fetching pages for years to power search across Apple's ecosystem, and most sites have simply never had a reason to think about it, which means most robots files inherited a permissive default that predates anybody caring about generative training.

That is exactly why it is worth a deliberate decision now rather than a default. If you have written a Google-Extended rule on principle, the same principle almost certainly applies to Apple, and a policy that refuses one company's foundation-model training while silently permitting another's is not a policy — it is whichever documentation somebody happened to read.

The reverse holds too. If you concluded that generative distribution is good for your business and left Google-Extended alone, then leaving Applebot-Extended alone is the consistent choice, and it is worth writing that down so the next person to audit the file does not "fix" it.

Writing a file that survives its author

Because both tokens are unobservable and both sit one hyphen away from a token that carries your search traffic, this is the corner of a robots file that most needs to explain itself.

Comment every block with what it is and what it is not. A line saying that this token controls AI training only and does not affect search inclusion, with a link to the operator's page that says so, takes ten seconds to write and prevents the entire class of accident this page is about. The next person to open the file will be under time pressure, will not have read either operator's documentation, and will pattern-match on the names.

Keep the parent-crawler rules physically separate from the extended-token rules — different sections, with a comment between them — so that a careless edit to one is visually obvious rather than a one-character diff in the middle of a block.

And treat the file as code. It belongs in review, it belongs in version control, and a change to it deserves the same scrutiny as a change to routing. A robots.txt diff is one of the few one-line changes that can remove a company from search, and it is routinely the least-reviewed file in a repository.

The dependency people miss

An extended token only governs content the parent crawler actually fetched. If Applebot cannot reach a page, Applebot-Extended has nothing to govern on it. If Googlebot is blocked from a directory, the Google-Extended rule for that directory is decorative.

This matters in the common case where a site has accumulated rules over years. A legacy Disallow on a documentation path, added for reasons nobody remembers, silently makes both the search decision and the AI decision for that path — and the AI decision looks like it was made deliberately in the robots file when it was actually made by accident a long time ago.

So the audit order is: parent crawler first, extended token second. Confirm what Applebot and Googlebot can actually reach, on every host you serve, and only then reason about the usage tokens.

What you can measure and what you cannot

Be clear with yourself and with anyone you report to. The extended tokens are unobservable. Any dashboard claiming to show Applebot-Extended or Google-Extended activity is showing you something that does not exist.

What is observable, and worth watching, is the parent crawlers. Applebot and Googlebot both send real requests with real user agents, and whether they are reaching your important pages is a question with a factual answer. Attensira's crawler logs record fetches by agent and URL, which covers the half of this picture that can be known. The bot access score will read back what your current rules permit before you edit anything, and the robots.txt generator will assemble a file that distinguishes the parent tokens from the extended ones correctly.

Related comparisons

For Google-Extended set directly against the crawler it depends on, read Google-Extended vs Googlebot. For the alternative design — a separate crawler for training rather than a usage token — see GPTBot vs OAI-SearchBot and Meta-ExternalAgent vs GPTBot. The open-archive route that no vendor token reaches is covered in CCBot vs GPTBot.