The AI Read is on a break. The last edition was November 15, 2026. Everything here stays open to read.
The AI Read
← AI News

Illustration, AI-generated from the story text. Tap a panel to read the story through it.

Afternoon Brief · November 8th 2026

AI auditors catch most agent misbehavior, but at 99% detection the best one flags 15% of safe runs

Andy Hall, a researcher at the Anthropic Institute on leave from Stanford, tested five models as auditors of agent logs, writing in a personal capacity. On METR's reward-hacking cases, GPT-5 caught nearly 98% at a 5% false-alarm rate, at about an eighth of GPT-6 Astra's cost, and outscored it. Gemini 3.8 Flash caught about half. On sandbagging, all five caught 71% to 81%. Pushed to 99% detection on a separate set of unsafe actions, the best model, GPT-6 Astra, flagged 15% of safe ones, and Gemini 3.8 Flash flagged 81%.

The arithmetic is the finding. At 100,000 sessions a month, a 3% false-alarm rate produces nearly 3,000 false warnings against about 2,000 real ones, and Hall expects human reviewers to stop reading alerts. Newer, costlier models did not reliably audit better, and Fable 5 and Opus 5 refused to run the evaluation. The reward-hacking set is only 47 cases, so one case moves the rate by about two points. SourcesBA

False alarms when an auditor is pushed to 99% detection
Share of safe agent actions flagged, R-Judge set
Gemini 3.8 Flash81%GPT-6 Astra (best)15%
Andy Hall pilot study, November 2026. [A]

Sources

This story ran in the Afternoon Brief of November 8th 2026.