AI8 min readBy JH Akash
One robots.txt Rule to Opt Out of AI Training Explained
On Oct 30, 2026, OpenAI, Google, Microsoft and Anthropic backed one robots.txt token to opt out of AI training. Dates, limits and the exact code.
On October 30, 2026, four of the biggest AI labs did something unusual: they agreed on the same rule. OpenAI, Google, Microsoft and Anthropic all said they will honor one shared robots.txt token, _usageai-training, that lets a website opt out of AI training in a single step. Until now, a publisher who wanted out had to block each company's crawler by name, GPTBot, Google-Extended, ClaudeBot, and a growing list of others. This announcement collapses that chore into one entry.
The catch is in the timing and the fine print. Only Anthropic says it has already started implementing support. Microsoft starts on February 1, 2027. Google aims for 2027. OpenAI says "in the coming months" with no date. And the signal is voluntary, it covers training only, and it does nothing about content that has already been collected or trained on. So this is a real coordination step, not an instant off switch.
The quick verdict
| Question | Answer |
|---|---|
| What was announced? | One shared robots.txt policy token, _usageai-training, to opt out of AI training |
| Announced | October 30, 2026 |
| Backed by | OpenAI, Google, Microsoft and Anthropic |
| The code | User-Agent: _usageai-training, then Disallow: / |
| Draft behind it | IETF Internet-Draft "Machine-Readable Training Preference Signal", by Kevin Kelley of Anthropic, dated October 30, 2026 |
| Draft status | An individual draft, not endorsed by the IETF, with no formal standing, expiring May 3, 2027 unless updated |
| What it covers | AI training only, not crawling, search, or AI answers |
| What it does not do | Remove content already collected or already trained into a model |
| Who can use it | Any website owner, by adding the token to robots.txt |
| What to keep for now | Your existing per-company rules, such as Google-Extended, because the shared token is mostly not live yet |
The exact code
Add this block to your robots.txt if you want to opt out of training on your whole site:
User-Agent: __usage_ai-training
Disallow: /To opt out of everything except one folder, the draft example adds an Allow line:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Two details matter. First, the token must match exactly, apart from letter case. A generic User-Agent: * group does not count, and if no group names the token, the site has expressed no preference. Second, you can opt out of only some folders instead of the whole site, by using Disallow on the paths you want to protect.
Who honors it, and when
| Company | What it says | Timing |
|---|---|---|
| Anthropic | Has started implementing support for the shared token, alongside its existing controls | Already started, rollout across services takes time |
| OpenAI | Will honor a shared robots.txt signal for foundational generative AI training, existing crawler controls keep working | In the coming months, no date given |
| Microsoft | Will honor the training preference, plus a separate AI Grounding preference, and page-level controls | February 1, 2027, as pages are recrawled |
| Plans to treat an opt-out as if the publisher had disallowed Google-Extended, unless Google-Extended is set directly | Aims for 2027 |
Microsoft goes furthest. It is adding a page-level training setting and a separate grounding setting, which decides whether a page can feed live AI answers in Bing, Copilot and Microsoft 365. A third option, DisallowExtended, allows grounding in Microsoft's own products but not in the grounding services it sells to other companies. Microsoft also says a page can carry a Bingbot meta tag, ai-training:disallow, or an X-Robots-Tag header, which overrides robots.txt. Sites that rely on Bing's old NoArchive and NoCache tags should switch to the new preferences by March 1, 2027, after which the old tags are no longer read as a training or grounding preference.
Google's detail is worth reading carefully. If you use the shared token to opt out and you do not mention Google-Extended, Google plans to treat that as blocking Google-Extended. But if you explicitly allow Google-Extended, Google says that site remains eligible to show up and earn clicks through grounding in Gemini app answers. In other words, the opt-out choice and the visibility choice are linked, so set both deliberately.
Key dates, counted from the announcement
| Date | What happens | Days after October 30, 2026 |
|---|---|---|
| October 30, 2026 | Shared token announced by all four labs | 0 |
| February 1, 2027 | Microsoft starts honoring the new preferences | 94 |
| March 1, 2027 | Bing stops reading old NoArchive and NoCache tags as training or grounding preferences | 122 |
| May 3, 2027 | IETF draft expires unless it is updated | 185 |
| 2027, no month given | Google aims to support the token | Not dated yet |
| Coming months, no date given | OpenAI aims to implement the signal | Not dated yet |
The day counts are simple calendar arithmetic from October 30, 2026, not vendor claims. The two undated items, Google and OpenAI, are the reason publishers should not treat the announcement as live coverage today.
What it controls, and what it does not
| Question | Answer |
|---|---|
| Does it stop AI training on my content? | That is its purpose, once each lab implements it |
| Does it stop crawling? | No. The draft says the token is not a directive for crawling behavior, so bots may still fetch a page |
| Does it stop search indexing? | No. Microsoft says opting out of training does not affect eligibility for regular Bing results |
| Does it stop AI answers quoting my pages? | No, except through Microsoft's separate grounding control. Training and grounding are different uses |
| Does it delete past training data? | No. It only covers future training runs, not content already collected or already in model weights |
| Does it bind scrapers that ignore it? | No. It is voluntary, with no enforcement power, so non-compliant scrapers can disregard it |
This last limit is the one publishers keep running into. The token is a coordination mechanism, not a security control. It works only with companies that choose to follow it, and the draft itself has no formal IETF standing. That does not make it useless, but it does mean you should keep monitoring compliance crawler by crawler, and keep your existing controls in place until the shared token is actually honored.
What to do now
Add the token now, it costs nothing. The two-line block is already published, and adding it early means you are covered as support arrives, without waiting for each lab's rollout date.
Keep your existing rules. Google-Extended, GPTBot, ClaudeBot and your other per-company rules remain what works today. Do not delete them in favor of a token that is mostly not live yet.
Decide training and grounding separately. If you use Microsoft's products, set the training preference and the grounding preference as two choices, not one. If you care about Gemini app visibility, decide explicitly about Google-Extended instead of letting the shared token decide it by default.
Check Bing tags before March 1, 2027. If your site relies on NoArchive or NoCache, move to the new Microsoft preferences before that deadline.
Expect a follow-up draft. The current draft expires on May 3, 2027 unless it is updated, and the working group includes Meta, Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation and Creative Commons. The standard may change as those participants weigh in, so treat today's wording as a starting point, not a final law.
For teams building AI products, the lesson is broader than one robots.txt file. Training preferences, grounding preferences and search visibility are now three separate levers, and mixing them up costs either data control or traffic. CodeMyPixel builds AI agent development services and helps companies hire an AI automation agency to put AI into real workflows with the guardrails and evaluation that a raw model or a single robots.txt line does not provide. If you are weighing open models against closed ones, our Mistral Large 4 open weights breakdown and our Gemini 4 Carbon explainer show how access, licensing and control differ in practice.
Which to pick
Pick the shared token plus your old rules if you want maximum coverage today. The token prepares you for 2027, the old rules protect you now.
Pick training opt-out only if you want your content out of future model training but you still want search traffic and AI answer visibility. Remember that this token alone does not block AI answers.
Pick training plus grounding opt-out on Microsoft if you also want your pages out of live AI answers in Bing, Copilot and Microsoft 365. That is a separate Microsoft setting, not part of the shared token itself.
Pick no opt-out if your business benefits from AI training exposure, or if you want to stay eligible for Gemini app grounding and clicks, and you are comfortable with your content being used for training. This is a business choice, not a default, so make it deliberately.
What is the shared AI training opt-out token?
It is a robots.txt policy token, _usageai-training, announced on October 30, 2026, that OpenAI, Google, Microsoft and Anthropic say they will honor. A website owner adds User-Agent: _usageai-training followed by Disallow: / to opt out of AI training, instead of blocking each company's crawler by name.
Does opting out stop Google or Bing from indexing my site?
No. The token covers AI training only, not crawling or search. Microsoft says opting out of training does not affect eligibility for regular Bing results, and the draft says the token is not a directive for crawling behavior.
When does the shared token actually take effect?
Anthropic has already started implementing it. Microsoft starts on February 1, 2027. Google aims for 2027 with no month given. OpenAI says it will implement the signal in the coming months with no date given. Until then, your existing per-company rules are what works today.
Does the token remove my content from models already trained on it?
No. Updating robots.txt does not remove content already collected into datasets or already trained into model weights. The token only covers future training runs.
Is this a formal web standard?
No, not yet. It is an individual IETF Internet-Draft, "Machine-Readable Training Preference Signal", by Kevin Kelley of Anthropic, submitted to the IETF AI Preferences working group. The IETF datatracker notes that an individual draft is not endorsed by the IETF and has no formal standing, and the draft expires on May 3, 2027 unless it is updated.
Should my business opt out of AI training?
Opt out if controlling future training use matters more to you than any exposure that training might bring, and keep your old crawler rules until the shared token is live. Stay opted in, or set an explicit Google-Extended allow, if you want to remain eligible for Gemini app grounding and clicks. Decide training and grounding separately, and review the choice when Microsoft's February 1, 2027 date arrives.
- robots.txt
- ai training opt-out
- openai
- microsoft
- anthropic
- ai policy
