Illustration, AI-generated from the story text. Tap a panel to read the story through it.

1 / 4Ten of 25 popular AI benchmarks mismeasure what they claim, a difficulty audit finds×

The audit

A paper posted October 22nd by researchers at the University of Illinois treats each benchmark as a spread of question difficulty and audits 25 of them across seven domains using item-response theory, a method from standardized testing.

The gaps

Ten have a material mismatch with the even spread a neutral test needs, and eight have gaps large enough to hide real progress. The AIME 2024 and 2025 math sets are among the worst.

The fix

The authors' fix, BenchFill, rewrites existing questions to fill the gaps; three added items cut the largest gaps by 33% on average.

What it means

It is a concrete reason to distrust small score moves on headline math tests.

Sources and the full story

‹›

Ten of 25 popular AI benchmarks mismeasure what they claim, a difficulty audit finds

A paper posted October 22nd by researchers at the University of Illinois treats each benchmark as a spread of question difficulty and audits 25 of them across seven domains using item-response theory, a method from standardized testing. Ten have a material mismatch with the even spread a neutral test needs, and eight have gaps large enough to hide real progress. The AIME 2024 and 2025 math sets are among the worst.

The authors' fix, BenchFill, rewrites existing questions to fill the gaps; three added items cut the largest gaps by 33% on average. It is a concrete reason to distrust small score moves on headline math tests. SourcesA

Sources

• AYour benchmark is an empirical measure over difficulty (Hugging Face Papers)huggingface.co

This story ran in the Morning Brief of October 31st 2026.