Google says it plans to support a shared, cross-industry control in robots.txt that would let website owners opt out of AI training by several AI providers with one instruction. The plan was announced on the Google Search Central blog on 30 October 2026, in a post by Google Search Advocate John Mueller.

The control is based on work by the AI Preferences (AIPREF) working group at the Internet Engineering Task Force (IETF). Google calls it a "draft proposal" and says it aims to support the control "along with other AI providers in 2027".

Background

robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may access. Google's AI training control so far has been Google-Extended, a robots.txt product token that the company says it has offered since 2023. According to Google's crawler documentation, Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps. The same page states:

“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

Google crawler documentation, last updated 30 October 2026

How the control would work

Google says that, "Instead of managing separate instructions for every AI developer", site owners will be able to use a shared "policy token" in robots.txt to set broad preferences across participating AI providers.

“When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended.”

John Mueller, Google Search Central Blog, 30 October 2026

Google says other AI providers "can handle it similarly". Company-specific controls such as Google-Extended will keep working alongside the shared signal. Google adds that websites which specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

The blog post gives no example syntax. Google's crawler documentation carries a note that Google "intends to support the AIPREF shared machine-readable training preference signal (with user-agent: __usage_ai-training) in early 2027."

The wider context

Google says the proposal reflects feedback from "publishers, civil society groups, regulators and other stakeholders" who have called for industry-wide controls on AI training. It says the approach is consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act, which it says "recognizes robots.txt as a primary method for web publishers to express machine-readable preferences."

The AIPREF working group's charter covers a vocabulary for AI-related preferences and ways to attach them to content, including through the Robots Exclusion Protocol (RFC 9309) and HTTP response header fields. It lists "technical enforcement of preferences" as out of scope. Google says discussions continue "on the rest of the AIPREF vocabulary" and that it will keep working to "refine and standardize this through the IETF working group."

What this means for site owners

Analysis by Depakar B

Nothing changes yet. Google's support is planned for 2027, and a shared token is only useful once other AI providers honour it too. Until then, Google-Extended is the training control Google documents.

Read the details closely when the token arrives. Google links Google-Extended to grounding in Gemini Apps as well as training, so a blanket opt-out could also affect whether your pages appear and earn clicks there. Decide what you want before you copy a rule from a forum.

On WordPress, robots.txt is often not a file on the server. WordPress can generate one itself through its do_robots() function, and SEO plugins often manage it too. Find out where yours comes from before you edit it, then check that the live /robots.txt shows your change. For the WordPress how-to on adding __usage_ai-training safely, see Robots.txt AI training opt-out for WordPress. My explainer on Google's AI Overviews controls covers the related Search-side options.

About this report

This report is based on Google's Search Central blog and crawler documentation, and the IETF AIPREF working group charter. Facts and quotes are taken from the sources credited below, which I re-read on 3 November 2026. Sections marked "Analysis" are my own view. I am an independent WordPress developer and am not affiliated with the organisations named here.

Credits and sources

Corrections

Updated 5 November 2026, 11:40 IST:52 IST: the opening of "How the control would work" now attributes the separate-rules point to Google, the robots.txt analysis now cites the WordPress code reference, and a background section, quotes and full source credits were added.

Spotted an error? Please contact me. Corrections are noted on this page with the date.