Four different spider-like crawler robots approach a website's front gate, where a single lever on the gatepost has been pulled down.

Outside the gate, the four robots have turned their backs on the field of books and are walking away down the road at different paces: one already far off, one mid-turn, two close behind; none goes through the gate.

Through the same unguarded gate, a robot carries a single page out to a person waiting at a desk, while the lever stays down and no lock or guard is anywhere.

Illustration, AI-generated from the story text. Tap a panel to read the story through it.

Afternoon Brief · October 30th 2026

OpenAI, Google, Microsoft and Anthropic back one robots.txt switch to refuse AI training

The four companies have said they will honor a single "policy token" in a site's robots.txt that opts it out of AI training across all their crawlers, instead of a separate block for each one. The signal is an IETF draft, draft-kelley-aipref-training-preference-signal, submitted by Kevin Kelley, a standards lead at Anthropic. Timing varies by company in the reports: Anthropic is described as already honoring it, OpenAI says "coming months," and Google and Microsoft point to next year. Google says a site that uses the token to refuse training will be treated as having blocked Google-Extended, and Microsoft says the choice will not affect Bing rankings. It covers training only. Grounding, search answers and agents fetching a page for a user are outside it, and there is no enforcement mechanism beyond the crawlers' own compliance. SourcesB C

Sources

This story ran in the Afternoon Brief of October 30th 2026.

Binance loses $6.3 billion in the largest crypto theft on record, laundered by what looks like automation

Britain's competition regulator investigates Meta's terms for using publisher content in AI