This guide explains what the new __usage_ai-training rule actually says, what it does not do, when the companies say they will start reading it, and how to add it to WordPress safely. Facts below were checked against the original announcements and documentation on 10 November 2026. Dates and support are still moving, so treat this as a snapshot rather than a final standard.
What Was Announced on 30 October 2026
Three announcements landed on the same day, all pointing at one Internet-Draft:
- The IETF draft. “Machine-Readable Training Preference Signal” (draft-kelley-aipref-training-preference-signal-00) was posted by Kevin Kelley on 30 October 2026. It defines one policy token,
__usage_ai-training, that you place in an ordinary robots.txt group. The datatracker notes that an Internet-Draft has no formal standing in the IETF standards process, and this one is version -00. - Google. John Mueller’s Search Central post says Google plans to support the shared control, and Google’s crawler documentation says it intends to do so in early 2027. If the shared token opts out and Google-Extended is not explicitly specified, Google plans to treat it as if you had disallowed Google-Extended.
- Microsoft and OpenAI. Bing’s post by Krishna Madhavan says Microsoft will start honouring the new preferences as it crawls from 1 February 2027. OpenAI says it will honour the common signal, aims to implement it “in the coming months”, and that its existing crawler controls will continue to apply.
The Verge also reported that Anthropic, where the draft’s author works, intends to adopt it. I have not found a separate Anthropic announcement, so I would not rely on a date for that yet.
How the Robots.txt AI Training Opt-Out Rule Works
The draft adds no new directives and no new grammar. You write a normal group, but the User-agent line names a purpose instead of a crawler:
User-agent: __usage_ai-training
Disallow: /
Allow: /press/
Under the draft, a disallowed path carries a preference against AI training and an allowed path carries a preference in favour. That example asks participating providers not to train on anything except the /press/ folder. Path matching and longest-match precedence follow the existing robots.txt standard, RFC 9309, so the rules behave the way you already expect.
Three details matter more than the headline:
- The wildcard does not count. The draft says
User-agent: *groups must not be read as matching the token. Your existing catch-all rules express no training preference at all. - No group means no preference. If the file has no
__usage_ai-trainingline, nothing has been said. Microsoft’s post is explicit that “Allow” is the default when no training preference is published. - It is not a crawl directive. The draft says the token does not control crawling. Disallowing it will not stop Googlebot, Bingbot or GPTBot fetching pages.
What the Shared Rule Does Not Change
The biggest risk is assuming this rule does more than it does. It is a preference about training, read only by providers that choose to honour it. It is not a privacy wall and it is not an AI search switch.
- Google Search is unaffected. Google’s documentation says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. The shared token maps to that same control.
- Bing search is unaffected. Microsoft says these usage preferences do not change eligibility for traditional Bing search results.
- Grounding is a separate choice. Google-Extended also covers grounding in Gemini Apps, and Google notes that sites which explicitly allow Google-Extended stay eligible to appear and receive clicks there. Microsoft has a separate page-level AI Grounding preference with no site-wide robots.txt equivalent.
- Bad bots will not care. Robots.txt is voluntary. Scrapers that ignore it today will ignore this group too.
If your real worry is AI Overviews rather than model training, that is a different set of controls. My guide to AI Overviews opt-out options covers what you can and cannot change there.
Decide Training and AI Visibility Separately
Before you touch the file, decide what you actually want. For most small business sites I work on, the answer is “stay visible everywhere people search, and think carefully about training”. You might reasonably decide either way on training. Being cited in AI answers can send visitors; training gives you nothing direct.
Work through the choices in this order:
- Search first. Never block Googlebot or Bingbot as a way to stop AI training. You would lose organic search, and the shared token makes that blunt approach unnecessary.
- Training next. Opt out with the shared token, opt out for some folders only, or leave it unset.
- AI answers last. Decide on Google-Extended for Gemini grounding, OAI-SearchBot for ChatGPT search, and Microsoft’s page-level grounding preference. OpenAI’s documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
If being cited is part of your plan, read how to get cited in AI Overviews and AI Mode before you opt out of anything.
How to Add It in WordPress
WordPress can serve robots.txt three ways, and you need to know which one your site uses before editing anything:
- A physical file. If a
robots.txtfile sits in the site root, the web server sends that and WordPress is not involved. - An SEO plugin editor. Many SEO plugins offer a robots.txt editor that writes either a physical file or filtered output.
- The virtual file. With no physical file, WordPress generates one through
do_robots(), and developers can change it with therobots_txtfilter.
Open https://yoursite.com/robots.txt in a private browser window and compare it with what your plugin shows. If they differ, a physical file or a host rule is winning. Then add the new group at the end, after your existing rules:
# Shared AI training preference (IETF draft, Oct 2026)
User-agent: __usage_ai-training
Disallow: /
On a custom build where you prefer code, a small filter in a site plugin does the same job for the virtual file:
add_filter( 'robots_txt', function ( $output, $public ) {
if ( $public ) {
$output .= "\nUser-agent: __usage_ai-training\nDisallow: /\n";
}
return $output;
}, 20, 2 );
Do not stack the token under User-agent: *. Consecutive User-agent lines share one group in robots.txt. If you put __usage_ai-training directly below User-agent: * and then write Disallow: /, you have also told every search crawler to leave. Give the token its own block with a blank line before it.
Keep Your Existing Bot Rules for Now
None of the three companies says the shared token replaces its own controls. Google says company-specific controls “will continue to function alongside this new joint signal”. OpenAI says its existing crawler controls continue to apply. Microsoft is replacing its older NoArchive and NoCache handling for training and grounding, which is a separate change.
So for the next few months:
- Leave any existing
Google-ExtendedandGPTBotgroups in place. Today, they are the rules that actually get read. - If you explicitly allow Google-Extended while disallowing the shared token, Google says the specific rule applies. That combination keeps Gemini grounding eligibility while opting out elsewhere.
- If your site uses
NoArchiveorNoCachefor Bing, read Microsoft’s transition steps. It asks you to publish new preferences before 1 March 2027, with a temporary grounding setting in Bing Webmaster Tools from 15 December 2026.
Bing also notes that publishing a preference is not the same as activating it. A page has to be recrawled before the setting is discovered.
Test the File After Every Change
Robots.txt mistakes are quiet. Nothing breaks visibly; traffic just drifts down weeks later. After you publish the new group:
- Fetch the live file and confirm the new block sits on its own, separated by a blank line.
- Check that
User-agent: *still has exactly the rules it had before. - In WordPress, confirm Settings › Reading does not have “Discourage search engines from indexing this site” ticked.
- Open the robots.txt report in Google Search Console and check the latest fetched version and any errors.
- Note the date and reason in your maintenance log so the next person does not undo it.
Robots.txt is one line in a larger technical picture. My WordPress technical SEO checklist covers the rest, and the llms.txt guide explains why that separate file is not an access control either.
Frequently Asked Questions
What is __usage_ai-training in robots.txt?
It is a policy token defined in an October 2026 IETF Internet-Draft. A robots.txt group with that token as its User-agent states your preference about AI model training for the paths it covers. Disallow means you prefer no training; Allow means you are in favour.
Will adding the shared AI training rule hurt my Google rankings?
Not if you add it as its own group. Google says Google-Extended, which the token maps to, does not affect inclusion in Google Search and is not a ranking signal. Problems only start if you merge it with User-agent: * or block Googlebot itself.
Does the shared rule work right now?
No provider has said it reads the rule yet. As of 10 November 2026, Google says early 2027, Microsoft says from 1 February 2027, and OpenAI says in the coming months. Publishing it now is harmless and means it is in place when support starts.
Should I remove my GPTBot and Google-Extended rules?
Not yet. Google and OpenAI both say their existing controls continue to work alongside the shared signal, and they are the rules being read today. Review them again once each provider confirms its implementation.
Does opting out of AI training remove my site from AI answers?
Not by itself. Training and answer use are separate choices. Google-Extended also covers Gemini grounding, OpenAI uses OAI-SearchBot for ChatGPT search, and Microsoft has a separate page-level grounding preference.
Final Thoughts
The shared robots.txt AI training opt-out is a sensible simplification: one rule instead of a list that never stays current. It is also a draft, not a finished standard, and nothing reads it yet. Add it as a separate group, keep your existing bot rules, protect your search crawlers, and recheck the file once Google, Microsoft and OpenAI confirm their implementations in 2027.
For the original announcement and what Google said it will support, see my news report: Google plans to honour a shared AI training opt-out in robots.txt.
If you would like someone to review your robots.txt, AI crawler settings and wider technical SEO in one pass, my SEO improvement service covers exactly that kind of audit.
Official References
Sources checked on 10 November 2026:
- Google Search Central Blog: Supporting a shared AI training control for websites (30 October 2026)
- Google Crawling Infrastructure: Google’s common crawlers (Google-Extended)
- IETF Datatracker: draft-kelley-aipref-training-preference-signal-00
- Microsoft Bing Blogs: Your content, your choice: clearer preferences for training and grounding
- OpenAI: Making it easier for websites to express AI training preferences
- OpenAI Developers: Overview of OpenAI crawlers
- WordPress Developer Resources: robots_txt filter
- The Verge: The biggest names in AI are backing a new way to opt out of scraping