OpenAI, Google, Microsoft and Anthropic back new opt-out for AI scraping
The four firms say they will support a shared signal in robots.txt that publishers can use to block AI training across companies.
Key facts
Story- Four major AI firms say they will adopt a common opt-out signal for scraping
- Publishers could set one AI training preference that applies across several companies
- The signal, called a policy token, is proposed through the IETF standards body
What's new
OpenAI, Google, Microsoft and Anthropic have said they will adopt a new standard that lets publishers opt out of being scraped for AI. The Verge reports the move, which was first reported by Search Engine Roundtable. Engadget says the four firms pledged to support a common signal in robots.txt that blocks scraping for AI training.
Why it matters
Today, publishers must set AI training preferences company by company. A single shared signal could make that far simpler. Website owners in India who publish news, reviews or other content could manage their choices in one place. Their work would be less likely to feed AI models without their say.
What others say
The Verge says the signal, called a "policy token", was put forward by Kevin Kelley, who leads technical standards work at Anthropic, within the Internet Engineering Task Force (IETF). The signal would sit in robots.txt, the file that tells crawlers which parts of a site they may visit. Engadget frames the news as Google joining OpenAI and Anthropic in promising new scraping controls.
What we don't know yet
The sources do not say when the standard will be finalised or put into use. They also do not say how strictly each company will honour the signal.
Worth your time?
One tap helps idly bring you more of what matters.
Advertisement
