What the New AI Training Opt-Out Really Means

Alissa Cooper is the executive director of the Knight-Georgetown Institute (KGI).

Last week, Google, Microsoft, Anthropic, and OpenAI all announced plans to support a new mechanism that lets web publishers express their desire to opt out of having their content used to train AI models. The proposal, authored by Anthropic’s Kevin Kelley, attracted considerable attention across the tech press (see here and here) and on social media, with the coverage highlighting the prospect of giving publishers greater control over how AI companies use their content.

But the announcements — and much of the subsequent coverage — leave out important context about the relationship between the product changes, ongoing standardization activity, and the policy environment. Understanding these nuances requires looking at the ongoing standards process and the broader policy debates surrounding AI, search, and competition.

The new proposal comes after years of disputes between publishers and companies that crawl the web — in court rooms, industry standards bodies, and public forums. Nearly two years ago, the Internet Engineering Task Force (IETF), the premier internet standards development organization, launched an effort known as AI Preferences (“AIPREF”) to develop new standards that would, among other things, give websites expanded options for automatically expressing preferences about how their content gets used for AI-related purposes. This work is meant to build in part on a prior IETF standard, RFC 9309: Robots Exclusion Protocol, which specifies the format for the commonly used “robots.txt” files employed by websites to control crawlers. Participants from Anthropic, Google, and Microsoft have all been actively involved in the AIPREF group, while OpenAI contributions have been minimal, despite the company stating last week that the announcement “strengthens our continued participation in the IETF standards discussions.”

This AIPREF work was sparked in part by the process of developing the EU General-Purpose AI Code of Practice, a voluntary compliance mechanism under the EU AI Act. Is signatories (most of the largest American and European AI model companies) commit to abide by the machine-readable preferences that sites express.

When looking beyond the headlines, what’s most important to understand about these announcements?

1. The mechanism is not an official “standard” — but adoption by the four largest search and AI providers might make it a de facto one.

Last week’s proposal has no official status in the IETF. It will need to be discussed and debated (probably at some length) to see if it will move forward toward standardization. But when the largest companies deploy support for it (all have committed to do so by 2027), that standardization status may not matter. If site operators are convinced to change their robots.txt files to comply with the new proposal, it may become the de facto way of signaling AI training preferences regardless of where the standards end up.

The proposal announced on October 30 (the “policy token draft,” for short) is documented in what is colloquially known as an “individual draft” — a submission to the IETF made by an author, which has no formal standing of any kind in the standards process. Anyone can submit an individual draft to the IETF, and over 1,000 are submitted each year.

The AIPREF working group already has two other drafts that are further along in the standards process, one that expresses a vocabulary for AI-related preferences (the “vocabulary draft”) and one that specifies how those preferences could be expressed in the widely used robots.txt files that many websites already publish (the “attachment draft”). In IETF parlance, these two drafts have been “adopted” by the AIPREF working group, which means the participants in the working group had rough consensus to pursue those approaches to meet the working group’s goals.

The policy token draft has no such consensus since it’s one individual’s submission that has not yet been considered by the group. After discussion, the group may or may not adopt it or integrate its contents into the attachment draft.

2. A training preference is nice to have, but a search-specific preference is missing, while the spotlight in the industry has shifted to inference.

Much of the discussion in the AIPREF working group over the last two years has centered on the definitions of new AI-related preferences in the vocabulary draft. These definitions matter because they are the categories for which sites will be able to express preferences in their robots.txt files (and via other technical mechanisms).

After significant discussion, the group has made progress towards settling on a definition of “AI training,” although discussions of some wording remain ongoing. After significantly more (and more heated) discussion, a definition of what it means for website content to be used in “search” has also been (mostly) provisionally agreed. Both definitions saw updates and new text added after the working group met in person at the end of August. Contentious discussions about a broader and more amorphous category — which has variously tried to cover “AI use,” “inference,” “grounding,” and related uses that occur separate from model training — are ongoing.

The policy token draft specifically leverages the “AI training” definition from the vocabulary draft and proposes a way that sites could express a preference to disallow all AI training, across all crawlers, by adding a few simple lines to their existing robots.txt files. The attachment draft already contains a similar mechanism, but the advantage of the approach in the policy token draft is that it would more immediately be machine-readable by many crawlers that are already deployed. This is the key value of the approach in the policy token draft: for site owners suffering from having to constantly update an ever-growing list of crawlers that they individually disallow, the newly proposed mechanism offers the prospect of a single, simple solution to express a preference to disallow all crawlers from AI training.

But what about search uses? And inference-time uses more broadly?

It is notable that the policy token draft does not provide an equivalent mechanism to cover search, especially given that the two companies that represent almost the entire web search market have signed up already to support it. It could have been quite powerful and clarifying for them to send the message that they want to make “search: yes; AI training: no” simple for sites to declare. The search definition in the vocabulary draft could have been leveraged in the same way that the AI training definition was. Publishers’ inability to express search-specific preferences has been the subject of regulatory investigation and intervention across multiple jurisdictions, making this omission all the more striking.

The new proposal’s focus on training also reinforces the difficult line-drawing that is still to come around inference-time uses more broadly. While the AIPREF working group has churned through multiple attempts at defining an inference or “AI usage” category, consensus has remained elusive, even as negotiation of licensing deals has shifted towards inference and recurring revenue. Microsoft’s announcement indicated that the company will combine support for the policy token mechanism with its own company-specific grounding preference mechanism, demonstrating the need for preference signals that go beyond training.

3. Since the mechanism has yet to be standardized, signatories to the EU AI Act Code of Practice are not required to commit to recognizing it.

Google’s announcement stated that the proposed mechanism is “consistent with the EU Code of Practice for General Purpose AI (GPAI) Model Providers under the EU AI Act.” Yet the text of the Code of Practice is clear that it commits signatories to follow standardized crawler instructions only:

Signatories commit … to employ web-crawlers that read and follow instructions expressed in accordance with the Robot Exclusion Protocol (robots.txt), as specified in the Internet Engineering Task Force (IETF) Request for Comments No. 9309, and any subsequent version of this Protocol for which the IETF demonstrates that it is technically feasible and implementable by AI providers and content providers, including rightsholders

As noted above, with the policy token draft having been published less than a week ago, it has a ways to go before being formally standardized, extending RFC 9309, or demonstrating technical feasibility. As much as all involved surely want a speedy resolution, giving interested stakeholders a chance to review and debate the mechanism’s design is a critical part of the process that helps yield stronger standards that are more easily deployable and (hopefully) meet the needs of more website owners.

Conclusion

A common mechanism supported by the largest AI and search providers could substantially reduce the burden on publishers who must currently manage crawler rules one crawler at a time. But it is important to be clear about the limitations of what has been announced: the preference applies only to AI training, leaving out search and other inference-time uses. As an individual draft, it has many steps to go before formal standardization, during which time its design could change significantly. That process should be allowed to run before a few of the world’s largest companies decide the rules for the wider web to have to follow.