Study finds LLM research claims age unevenly across models

Stanford researchers Owen Queen and James Zou tested 50 widely cited findings about large language models against 20 models and classified 44 percent as expired, meaning the original effect had largely disappeared. Another 34 percent had weakened, according to the-decoder.com.

The models came from eight developers and were released between July 2024 and September 2026. Among the 22 findings with an estimated expiration date, the average lifespan was 15.7 months. That figure describes those findings, not a shelf life that readers can apply to every AI study.

For teams choosing a model, the more useful result is the variation within a release window. Among models released between July and September 2026, expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Calculated from those counts: the difference is 19 findings, or 38 percentage points of the 50-finding set: (30 − 11) ÷ 50 × 100. This measures how differently the models reproduced earlier findings. It is not an overall quality score, but it shows why a shared release period cannot establish shared behavior.

A concrete selection decision follows: choose evidence that matches the intended task before using a broad model comparison. For a scientific-reference workflow, the reported citation experiment is directly relevant. Using prompts from an earlier study, GPT-6.1 Sol fabricated zero percent of references, Claude Sonnet 5.5 fabricated 1 percent, and Qwen3.7-Flash fabricated 69 percent. Those results support different judgments about these models on that experiment. They do not establish accuracy on other reference requests.

The same evidence also argues for separating reference accuracy from reasoning accuracy in an acceptance decision. Claude Sonnet 5.5 still failed about 10 percent of two-hop questions where it answered both component questions correctly. A workflow that retrieves references and then combines their findings therefore has two distinct acceptance questions: are the references real, and does the conclusion follow? Passing the reported citation test cannot answer the second.

For teams using models to judge submissions or other model outputs, a different finding matters: preference for a model's own answers was among the behaviors that persisted or strengthened. That makes an evaluator's identity relevant to the decision. Include comparisons between its own outputs and others' outputs against independently assigned judgments before allowing its scores to determine selection. Improved citation performance provides no evidence that this evaluator preference has disappeared.