Major AI Companies Back Shared Website Signal to Opt Out of Scraping
publishers 118articles 393first reported 23 Sep, 14:00 UTCdeveloping since 30 Sep · 33 editions
OpenAI, Google, Microsoft and Anthropic have committed to supporting a shared signal that lets website publishers communicate AI scraping preferences to multiple companies through a single setting.
The proposed “policy token,” developed by Anthropic standards lead Kevin Kelley as part of the Internet Engineering Task Force, would be placed in a site’s robots.txt file. That file already tells web crawlers whether they may access content for purposes such as search indexing, archiving and AI training. Publishers currently must communicate their AI preferences separately to individual companies.
The companies said their AI crawlers would not access a site when they encounter the shared opt-out token. Anthropic has already adopted the signal, while OpenAI plans to implement it in the coming months. Google and Microsoft expect to add support next year.
Microsoft said publishers will be able to distinguish between AI training and AI grounding, in which models are directed to verified information. A site’s choices will not affect whether it appears in Bing search results. Google has previously introduced controls separating participation in AI features and training from ordinary search visibility.
Meta communications head Andy Stone expressed support for a common industry signal but did not say whether Meta would honor the setting.
HOW THIS STORY WAS MADE
Written from 8 articles, 6 with the publisher's own text; 8 independent newsrooms once syndicated copies count as one; this version written 750 h after the record first saw the story; the editor kept it as written.
- Publishers
- 118
- Source articles
- 393
- Given to the writer
- 8, 6 with the article's own text
- Independent newsrooms
- 8
- This version written
- 750 h after the record first saw the story
- Second model (editor)
- kept as written
- Publication gate
- passed
- Human review
- none
WHAT THE EDITOR DID
The second model checked the writer's draft against the same sources and published it as written.
Written by a language model from the sources above, then checked by a second model that may only cut, attribute or correct. How it works →