Illustration, AI-generated from the story text. Tap a panel to read the story through it.

Andy Hall, a researcher at the Anthropic Institute on leave from Stanford, tested five models as auditors of agent logs, writing in a personal capacity.

On 's reward-hacking cases, GPT-5 caught nearly 98% at a 5% false-alarm rate, at about an eighth of GPT-6 Astra's cost, and outscored it. Gemini 3.8 Flash caught about half. On , all five caught 71% to 81%.

Pushed to 99% detection on a separate set of unsafe actions, the best model, GPT-6 Astra, flagged 15% of safe ones, and Gemini 3.8 Flash flagged 81%. The arithmetic is the finding.

At 100,000 sessions a month, a 3% false-alarm rate produces nearly 3,000 false warnings against about 2,000 real ones, and Hall expects human reviewers to stop reading alerts. Newer, costlier models did not reliably audit better, and Fable 5 and Opus 5 refused to run the evaluation. The reward-hacking set is only 47 cases, so one case moves the rate by about two points.
AI auditors catch most agent misbehavior, but at 99% detection the best one flags 15% of safe runs
Andy Hall, a researcher at the Anthropic Institute on leave from Stanford, tested five models as auditors of agent logs, writing in a personal capacity. On METR's reward-hacking cases, GPT-5 caught nearly 98% at a 5% false-alarm rate, at about an eighth of GPT-6 Astra's cost, and outscored it. Gemini 3.8 Flash caught about half. On sandbagging, all five caught 71% to 81%. Pushed to 99% detection on a separate set of unsafe actions, the best model, GPT-6 Astra, flagged 15% of safe ones, and Gemini 3.8 Flash flagged 81%.
The arithmetic is the finding. At 100,000 sessions a month, a 3% false-alarm rate produces nearly 3,000 false warnings against about 2,000 real ones, and Hall expects human reviewers to stop reading alerts. Newer, costlier models did not reliably audit better, and Fable 5 and Opus 5 refused to run the evaluation. The reward-hacking set is only 47 cases, so one case moves the rate by about two points. SourcesBA
Sources
- BAI auditors catch most agent misbehavior but flood reviewers with false alarms (The Decoder)the-decoder.com
- AWe're using AI to audit AI (Andy Hall, Free Systems)freesystems.substack.com
This story ran in the Afternoon Brief of November 8th 2026.