Claim-Level Evaluation for AI Agents: Most Published Claims Do Not Survive Their Own Data
AI agents are compared by scores, but what papers, leaderboards and model launches publish are claims: that a system is “#1”, better than a named rival, as good at lower cost, or on the cost–accuracy frontier. We propose claim-level evaluation: the unit of reporting is the claim, stated in one of a small set of testable forms and published with its bootstrap support proportion—the fraction of resampled datasets in which it holds. We give a test for each claim form and a one-command tool that applies it to any results file. To show why this is needed, we applied a protocol-bounded search to selected systems and sources, collecting 77 published claims from 51 public sources about systems on SWE-bench Verified, Terminal-Bench 2.0, τ²-Bench, and the Holistic Agent Leaderboard, and tested each against the per-task results it describes. Of the 43 claims about runs whose data are public, 29 (67%) fall below the 0.9 bootstrap-support threshold, and 6 are not even true of the published numbers. Rank claims fare worst: 2 of 21 “#1” claims are supported. This does not imply false reporting, but rather that benchmarks lack the statistical resolution to distinguish narrow margins: the unsupported leads have a median gap of only 0.4 points, requiring a median of 5.7× as many tasks to verify. Cost–accuracy frontier claims fare best (5 of 5), because costs differ by large factors. Seen at the claim level, SWE-bench Verified’s leader cannot be distinguished from 4 other entries and HAL’s τ-bench leader from 11; an unsupported lead would need a median of 5.7 times as many tasks to be supported. We release the claims, the coding rules, and the tool.
Authors
- Harish Mashetty
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24214791
- Primary Topic
- Scientific Computing and Data Management
- Type
- preprint