Two lines in robots.txt will be enough to tell several AI providers "don't train on my site" at once. On October 30, 2026, Google, Microsoft and OpenAI said they will honor the same token, __usage_ai-training, and Microsoft also put an end date on its current controls: from March 1, 2027, NoArchive and NoCache will no longer count as a preference for training or grounding.
What changes: one token per purpose, not per bot
Until now, opting out of training meant maintaining a list of names: Google-Extended, GPTBot, ClaudeBot… and Bing didn't even have a robots.txt token, only the NoArchive and NoCache tags. The new mechanism is defined in the Internet-Draft Machine-Readable Training Preference Signal, published on October 30, 2026 by Kevin Kelley of Anthropic. It reuses the RFC 9309 grammar unchanged: a group with a policy token on the User-Agent line that doesn't name a crawler, but a use of the content.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/This states a preference against training for the whole site, except /press-kit/. The draft borrows the IETF AIPREF working group's definition: training means using content to modify the parameters of a generative AI model; models that only classify or score are out of scope.
Each provider is rolling it out on its own schedule:
| Provider | When | How it reads the token |
|---|---|---|
| Microsoft (Bing, Copilot, Microsoft 365) | From February 1, 2027 | Training in robots.txt or per page; grounding separately, per page only |
| Early 2027 | A Disallow counts as blocking Google-Extended, unless you name it | |
| OpenAI | No date published | Will honor the shared signal |
Impact: what breaks if you do nothing
The shared signal is optional; Microsoft's change isn't. If your site relies on NoArchive or NoCache to keep Bing from training on your content or using it in generated answers, that protection now has an expiry date: from March 1, 2027, Microsoft will stop reading those tags as a training or grounding preference. And once it detects the new preferences on a site, it ignores the old ones there.
Microsoft is also splitting two uses that used to travel together:
- AI Training: changing the weights of future models. Declared in
robots.txt(site or path) or per page. - AI Grounding: using the page as context for a generated answer in Bing, Copilot or Microsoft 365. Per page only, with three values:
Allow(the default),DisallowExtended(Microsoft-operated products only, excluding Web IQ and API grounding in Foundry) andDisallow.
Neither affects eligibility for Bing search results, and both default to Allow when you publish nothing.
How to set it up
For the whole site or a path, use the shared robots.txt group; for individual pages, the Bingbot meta tag or response header that Microsoft documents:
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">X-Robots-Tag: Bingbot: ai-training:disallow
X-Robots-Tag: Bingbot: ai-grounding:disallowA page-level training preference overrides the robots.txt one. If you're coming from NoArchive or NoCache and want to stay out of grounding, Microsoft asks for two steps before March 1, 2027: publish ai-grounding on your pages, and record a temporary preference in Bing Webmaster Tools, which opens that option on December 15, 2026 for sites already using those tags. The temporary preference lasts until Bing recrawls the page or until September 30, 2027, whichever comes first.
If your site sits behind a CDN or manages headers as code (Nginx, Cloudflare, Firebase Hosting…), the header route is the easiest way to cover whole paths without touching templates.
Gotchas and edge cases
- The wildcard doesn't count. A
User-Agent: *group withDisallow: /expresses no training preference at all; the draft explicitly forbids reading it that way. No__usage_ai-traininggroup means you haven't said anything. - It doesn't block crawling. This is a usage preference, not a crawl directive. To limit traffic you still need per-bot groups.
- Specific beats general. Google only treats your shared
Disallowas aGoogle-Extendedblock when you don't nameGoogle-Extended; if you allow it explicitly, that rule wins. Scope matters too: per Google's crawler documentation,Google-Extendedalso governs grounding in Gemini Apps and Vertex AI. Opting out of training with the shared token takes you out of those answers as well, and allowingGoogle-Extendedto stay in them means letting Google train too. - Publishing isn't activating. Microsoft has to recrawl each page to pick up the preference; IndexNow can speed that up, but it doesn't guarantee a date.
- It's an individual draft. It expires on May 3, 2027 unless renewed, and has no formal working-group backing yet. What makes it useful is that the largest providers have already committed to reading it.
The concrete step for any site with original content: decide on training and grounding separately, add a __usage_ai-training group if you want to opt out of the former and, if you were counting on NoArchive with Bing, put March 1, 2027 in the calendar.
Sources
- Your content, your choice: clearer preferences for training and grounding, Microsoft Bing Blog, October 30, 2026.
- Supporting a shared AI training control for websites, Google Search Central Blog, October 30, 2026.
- Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), IETF, October 30, 2026.
- Making it easier for websites to express AI training preferences, OpenAI, October 30, 2026.
- Google's common crawlers, Google for Developers.