A Shared robots.txt Token for AI Training: Why User-Agent: * Says Nothing

On October 30, 2026, Google and Microsoft announced support for a single robots.txt signal that tells AI providers whether a site's content may be used for training. A site writes a group for User-Agent: __usage_ai-training and adds Disallow rules; an individual IETF draft posted the same day defines how that group is read. The line that matters most in the draft is that a User-Agent: * group must not be treated as matching the token. A site that blocks every bot, and a site with no rules at all, both express no training preference unless they add the token group. Microsoft starts honoring the setting on February 1, 2027; Google says early 2027.

This post covers what was announced, the result of applying the draft's lookup rules to five robots.txt files (including kortress.com's own), each company's timeline, Microsoft's separate grounding setting, and what changes for site owners.

What was announced

The announcement comes as three documents, and each does a different job.

Google says its existing company-specific controls keep working alongside the shared signal. If a site opts out with the shared token and does not name Google-Extended, Google plans to treat it as a Google-Extended disallow. That condition suggests an explicit Google-Extended rule would be followed instead, though Google does not state this precedence in so many words. Google also says the proposal is consistent with the EU Code of Practice for General Purpose AI model providers under the AI Act.

The draft's status needs to be clear. The IETF Datatracker marks it as an individual Internet-Draft that is "not endorsed by the IETF and has no formal standing in the IETF standards process." The author works at Anthropic, and the acknowledgments list participants from Google, Microsoft, OpenAI, Meta, Cloudflare and others, in their individual capacity. TNW and other outlets reported that OpenAI announced plans the same day, but OpenAI's own page returned HTTP 403 in the environment this post was written in, so we could not check it against the source. OpenAI's timeline is left out of this post for that reason.

How the draft reads a robots.txt file

The draft adds no new directives and no new grammar. It keeps the group structure and path matching of the existing robots.txt standard (RFC 9309) and puts a "policy token" that names a purpose, not a crawler, in the User-Agent line. The draft's own example:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

These three lines express a preference against training for the whole site, and in favor of it under /press-kit/. The lookup comes down to four rules.

1. If a path resolves to Disallow in the token group, that is a preference against training. Allow means a preference in favor.
2. A User-Agent: * group MUST NOT be interpreted as matching the token.
3. If the file has no User-Agent: __usage_ai-training line, no preference has been expressed. Company tokens like GPTBot keep their usual meaning but must not be treated as a shared preference.
4. The token is not a crawling directive. To limit crawling, you still write groups for crawler names.

Rules 2 and 3 make the biggest practical difference. The double-underscore prefix exists, per the draft, so that deployed parsers with looser matching are unlikely to match it by accident.

Five robots.txt files, checked

To see how the rules play out on real files, we implemented the draft's Section 4.2 lookup on top of RFC 9309 longest-match and ran it on five files. File 1 is kortress.com's actual robots.txt, downloaded on November 1, 2026 at 08:51 UTC. The other four mimic common setups.

Only files 2 and 5, which contain the token group, express a preference against training. File 3 blocks crawling outright but says nothing about training: under the draft, crawl rules and usage preferences are independent signals. File 4 blocks by company crawler name. Each named company's token keeps working as before, but other participants reading the shared token get no signal.

File 5 is where the shared result and Google's handling may split. By the shared token alone it reads as "against training". But Google says it will map a shared-token opt-out to Google-Extended only when Google-Extended is not specified. So this file would likely read as "no training for other participants, training allowed for Google." The draft leaves room for this: a company may document that it treats its own token as more specific than the shared one.

kortress.com currently expresses no training preference either way. Expressing one against training would take one extra group, with no change to the existing groups. This post does not recommend that setting; the diff below shows the difference between no preference and an explicit one.

 User-Agent: Bingbot
 Allow: /
+
+User-Agent: __usage_ai-training
+Disallow: /

Rollout by company

The announcements landed on the same day, but the dates they take effect do not. Figure 1 puts every date stated in the three documents and Google's crawler documentation on one axis.

Only Microsoft set dated steps. It begins honoring the new settings as it crawls from February 1, 2027, and after March 1 it stops reading NoArchive and NoCache as training or grounding preferences. Google's blog says it aims to support the control "along with other AI providers in 2027," and its crawler documentation says "early 2027." Neither gives a date. Draft -00 expires on May 3, 2027; unless a revision or a working-group document follows, the reference document's standing changes.

Microsoft's separate grounding setting

Microsoft adds a setting of its own on top of the shared training token. Grounding means using web content as input when an AI answer is generated. Training and grounding are independent, and both default to Allow when nothing is set.

DisallowExtended is the new option. It allows grounding in Microsoft-operated experiences such as Copilot and Microsoft 365, and excludes grounding services Microsoft provides to third parties, such as Web IQ and the grounding API in Foundry. Microsoft warns that restricting grounding reduces brand visibility in AI experiences, and states that neither choice guarantees inclusion, traffic, or payment. Neither setting changes eligibility for regular Bing search results, subject to normal crawl and indexing requirements.

What changes for site owners

Three judgments follow.

First, many sites that "already block AI" probably express no preference under this scheme. That includes sites that block with User-Agent: * (file 3) and sites that block individual company bots (file 4). If the goal is to opt out of training, the token group has to be written explicitly. Before the 2027 rollout, it is worth opening robots.txt and checking for the token line.

Second, training opt-out and visibility in AI answers are separate decisions. Microsoft separates them explicitly. Google says sites that allow Google-Extended are eligible to appear in, and receive clicks from, grounding in Gemini Apps. Because Google reads a shared-token opt-out as a Google-Extended disallow, opting out this way likely affects Gemini Apps grounding as well. A site that wants to refuse training but stay eligible there would, by Google's description, allow Google-Extended explicitly, as in file 5. Google did not show that combination as an example, so its documentation should be rechecked once support ships.

Third, the signal is voluntary and only covers participating companies. It applies to future training, not data already used, and the draft says private agreements may override it. For sites whose audience arrives mainly through Korean services such as Naver or Kakao, there is likely no immediate effect: neither company appears as an adopter in the Google or Microsoft posts or in the draft.

Even so, one signal for several companies is a clear improvement for site owners over chasing each company's crawler name. A similar move toward shared, vendor-neutral specs came up in our post on Agent Skills moving to the AAIF.

Community reaction

There is little reaction so far. Google's post was submitted to Hacker News as "Google plans to support shared AI training opt-out via robots.txt" and, when checked on November 1, 2026, had 7 points and no comments (HN 50309270). Reddit search could not be opened from the environment this post was written in.

Among trade outlets, TNW's October 31 article quoted draft author Kevin Kelley (Anthropic): "It'll take us some time to add support across our services…" (TNW). The same article, citing The Verge, reported that Meta communications head Andy Stone voiced support for a common signal on X without saying whether Meta would honor it. We found no named critics at the time of writing. The limits are stated in the sources themselves: Microsoft frames the setting around future training, and the draft says private agreements may override it.

Limitations

The results table comes from our own implementation of draft Section 4.2, not from Google's, Microsoft's, or Anthropic's parsers. Company-specific precedence (file 5) and differences in how deployed parsers match can change real outcomes.

Google's 2027 timing and exact behavior, and OpenAI's timeline, should be rechecked as each company updates its documentation.