Major AI Companies Back Shared Standard for Publishers to Opt Out of Scraping
publishers 118articles 391first reported 23 Sep, 14:00 UTCdeveloping since 30 Sep · 33 editions
OpenAI, Google, Microsoft and Anthropic have agreed to support a shared mechanism that lets website publishers state their AI scraping preferences once for multiple companies.
The proposed “policy token,” developed by Anthropic standards technical lead Kevin Kelley through the Internet Engineering Task Force, would be placed in a site’s robots.txt file. That file already tells web crawlers whether they may access content for purposes such as search indexing, archiving and AI training. The new signal is intended to replace the current practice of communicating preferences separately to each AI provider.
The companies said their AI crawlers will not access sites that use the token to opt out. Anthropic has already adopted the mechanism, while OpenAI plans implementation in the coming months and Google and Microsoft expect to add support next year.
Microsoft said publishers will be able to distinguish between AI training and AI grounding, in which models consult verified information. Sites opting out will remain eligible to appear in Bing search results. Google has previously introduced similar controls separating AI uses from search visibility.
Meta communications head Andy Stone expressed support for a common industry signal but did not commit Meta to honoring it. The initiative arrives as publishers seek greater control over AI crawlers, alongside measures including licensing standards and Cloudflare’s default blocking of such crawlers.
HOW THIS STORY WAS MADE
Written from 8 articles, 6 with the publisher's own text; 8 independent newsrooms once syndicated copies count as one; this version written 738 h after the record first saw the story; the editor kept it as written.
- Publishers
- 118
- Source articles
- 391
- Given to the writer
- 8, 6 with the article's own text
- Independent newsrooms
- 8
- This version written
- 738 h after the record first saw the story
- Second model (editor)
- kept as written
- Publication gate
- passed
- Human review
- none
WHAT THE EDITOR DID
The second model checked the writer's draft against the same sources and published it as written.
Written by a language model from the sources above, then checked by a second model that may only cut, attribute or correct. How it works →