The new robots.txt directive will prevent the companies from scraping content for generative AI training.
Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their content forgenerative AI training. The companies have decided to adopt a proposal under consideration at IETF, theopen standards organization for the internet, written by Anthropic technical lead for standardsKevin Kelley.
As OpenAI and Google explained in their announcements, website owners currently have to identify individual crawlers and add separate rules for each provider. Kelley’s proposal will change that by getting multiple AI companies to recognize and honor a single common signal.Publishers have to set up a new user agent in the robots.txt file that’s specifically for AI training, which will look like this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That will block crawlers specifically meant to gather data for AI training. The “Allow” section gives website operators the option to allow scraping for AI training for specific parts of the page, such as if there’s a press kit on it. But this new user-agent section doesn’t prevent crawling altogether or prevent content from showing up on Google search. For instance, Google will specifically interpret it as a stop signspecifically for the “Google-Extended” user agent, whichcrawls websites for materials used to trainGemini models.
Another thing to note is that this directive cannot undo any past scraping the companies have already done. If they’d crawled a page previously, they already have its content. This will just prevent them from scraping websites or specific pages in the future. Anthropic has already started supportingthe new directive, while Microsoft said it will start supporting iton February 1, 2027. Google said it will support it in early 2027, as well, while OpenAI pledged to implement the new signal in the coming months.