Claim-Level Evaluation for AI Agents: Most Published Claims Do Not Survive Their Own Data

AI agents are compared by scores, but what papers, leaderboards and model launches publish are claims: that a system is “#1”, better than a named rival, as good at lower cost, or on the cost–accuracy frontier. We propose claim-level evaluation: the unit of reporting is the claim, stated in one of a small set of testable forms and published with its bootstrap support proportion—the fraction of resampled datasets in which it holds. We give a test for each claim form and a one-command tool that applies it to any results file. To show why this is needed, we applied a protocol-bounded search to selected systems and sources, collecting 77 published claims from 51 public sources about systems on SWE-bench Verified, Terminal-Bench 2.0, τ²-Bench, and the Holistic Agent Leaderboard, and tested each against the per-task results it describes. Of the 43 claims about runs whose data are public, 29 (67%) fall below the 0.9 bootstrap-support threshold, and 6 are not even true of the published numbers. Rank claims fare worst: 2 of 21 “#1” claims are supported. This does not imply false reporting, but rather that benchmarks lack the statistical resolution to distinguish narrow margins: the unsupported leads have a median gap of only 0.4 points, requiring a median of 5.7× as many tasks to verify. Cost–accuracy frontier claims fare best (5 of 5), because costs differ by large factors. Seen at the claim level, SWE-bench Verified’s leader cannot be distinguished from 4 other entries and HAL’s τ-bench leader from 11; an unsupported lead would need a median of 5.7 times as many tasks to be supported. We release the claims, the coding rules, and the tool.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24214791
Primary Topic
Scientific Computing and Data Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Claim-Level Evaluation for AI Agents: Most Published Claims Do Not Survive Their Own Data

Harish Mashetty
Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
preprint

Claim-Level Evaluation for AI Agents: Most Published Claims Do Not Survive Their Own Data

Harish Mashetty
preprint en

Abstract

AI agents are compared by scores, but what papers, leaderboards and model launches publish are claims: that a system is “#1”, better than a named rival, as good at lower cost, or on the cost–accuracy frontier. We propose claim-level evaluation: the unit of reporting is the claim, stated in one of a small set of testable forms and published with its bootstrap support proportion—the fraction of resampled datasets in which it holds. We give a test for each claim form and a one-command tool that applies it to any results file. To show why this is needed, we applied a protocol-bounded search to selected systems and sources, collecting 77 published claims from 51 public sources about systems on SWE-bench Verified, Terminal-Bench 2.0, τ²-Bench, and the Holistic Agent Leaderboard, and tested each against the per-task results it describes. Of the 43 claims about runs whose data are public, 29 (67%) fall below the 0.9 bootstrap-support threshold, and 6 are not even true of the published numbers. Rank claims fare worst: 2 of 21 “#1” claims are supported. This does not imply false reporting, but rather that benchmarks lack the statistical resolution to distinguish narrow margins: the unsupported leads have a median gap of only 0.4 points, requiring a median of 5.7× as many tasks to verify. Cost–accuracy frontier claims fare best (5 of 5), because costs differ by large factors. Seen at the claim level, SWE-bench Verified’s leader cannot be distinguished from 4 other entries and HAL’s τ-bench leader from 11; an unsupported lead would need a median of 5.7 times as many tasks to be supported. We release the claims, the coding rules, and the tool.

Zenodo (CERN European Organization for Nuclear Research)
Scientific Computing and Data Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon