Illustration, AI-generated from the story text. Tap a panel to read the story through it.
Owen Queen and James Zou reran 50 widely cited findings across 20 models released from July 2024 to September 2026.
They class 22 as expired, 17 as weakened, 8 as persisting and 3 as stronger, and most expired or weakened within 15 months of publication.
The expired ones are mostly capability failures: counting letters, finding facts in the middle of long contexts, abandoning correct answers when challenged. In 2023 ChatGPT fabricated 55% of its citations; recent models rarely do.
Three grew stronger: judges favoring their own output, praise that follows a user's stated preference, and nonuniform random numbers. A claim about "LLMs" with no model and date attached tells a reader little.
Stanford retested 50 well-known findings about LLMs, and 44% no longer hold
Owen Queen and James Zou reran 50 widely cited findings across 20 models released from July 2024 to September 2026. They class 22 as expired, 17 as weakened, 8 as persisting and 3 as stronger, and most expired or weakened within 15 months of publication. The expired ones are mostly capability failures: counting letters, finding facts in the middle of long contexts, abandoning correct answers when challenged. In 2023 ChatGPT fabricated 55% of its citations; recent models rarely do. Three grew stronger: judges favoring their own output, praise that follows a user's stated preference, and nonuniform random numbers. A claim about "LLMs" with no model and date attached tells a reader little. SourcesA
Sources
- AThe shelf life of LLM findings (Queen and Zou, Stanford)zou-group.github.io
This story ran in the Afternoon Brief of November 3rd 2026.