Morning Brief, August 20th 2026

Editorial

The test nobody ran

Three of 1,357. Of all the AI medical devices the FDA has cleared for use on actual patients, three have been tested on whether patients do better with them. Not three percent. Three devices.

The mechanism that produced this number is worth taking apart, because it is not a scandal, it is a design. The 510(k) pathway clears a device by showing it is substantially equivalent to a device already cleared. That rule predates AI by decades and it made a rough kind of sense for scalpels and stents, where the physical object is the product and equivalence is inspectable. Applied to AI, it produces a chain: each model is cleared by resembling the last one, the last one was cleared by resembling its predecessor, and nowhere in the chain did anyone run the experiment the marketing implies, that care with the device beats care without it. The regulated claim is "reads an image like the previous device." The sold claim is "improves outcomes." The gap between those claims is 1,354 devices wide.

Here is why I think this is the most general story in the news and not a health-care curiosity. The same structure showed up twice more this week in places with no FDA at all. The Reconstruction benchmark found that frontier models, handed a bibliography and asked for the paper's idea, manage 3% to 15% alone, while a tournament of generators and judges reaches 42%: the capability was never in the generator, it was in the selection. Anthropic's binder results ran the same way, thousands of candidates, a filter, wet-lab verification at the end deciding what was real. Where verification is cheap and mandatory, compilers, test suites, protein assays, AI progress is fast and the claims hold. Where verification is expensive and optional, medicine, hiring, policing, the claims float free, and what fills the gap is equivalence: this model scores like that model, this tool is like the tool you already approved.

My position: the binding constraint on AI in high-stakes domains is no longer model capability, it is that nobody is paid to run the outcome trial. A trial costs millions and takes years; a 510(k) costs a filing. The vendor will not fund it, the buyer cannot, and the regulator does not require it. So the evaluation gap is not closing on its own, it is compounding, one equivalence at a time, and the first sector to industrialize outcome verification, not benchmarks, outcomes, will be the one where AI's value stops being arguable.

What would prove me wrong: the FDA's AI-device framework adding an outcome-evidence requirement with teeth within a year, or payers doing it first by refusing reimbursement without outcome data, which would work faster than regulation. Watch the reimbursement codes, not the clearances. If neither moves by next August, the three-of-1,357 ratio will still be roughly what it is today, and everyone deploying these systems will still be reasoning from a chain of resemblances that ends at a device nobody tested either.

Vera Lindqvist