Change in listed LMArena text ratings over the first 90 days after first listing, relative to established models: a pre-registered data note on 280 models first listed from September 2024 to August 2026
LMArena publishes ratings of large language models from pairwise human votes. A provider can test several private variants before a model is listed, and picking the best variant can inflate the first listed rating; LMArena has said that this bias goes to zero as fresh votes come in. We measured the change on LMArena's public snapshot dataset (lmarena-ai/leaderboard-dataset, CC BY 4.0, revision b9863c64, text leaderboard with style control, category overall, 198 snapshots from 2024-08-28 to 2026-11-18), under a plan whose rules were committed before any rating was read. For each of 280 models first listed after the first snapshot, D90 is the model's rating change from first listing to about 90 days later, minus the median change over models listed at least 180 days earlier. The mean D90 was -1.17 rating points (95% bootstrap CI -2.33 to -0.003; median -0.84). The interval lies inside the registered equivalence margin of plus or minus 5 points, so by the registered rule there is no practically relevant launch inflation in this cohort, in the sense the design measures: no mean fall larger than 5 points relative to established models within about 90 days. That supports "no large average fade", not a substantive null: the margin was set from per-model noise at listing, and the observed mean is what LMArena's 11-point figure would give if it applied, and fully faded, in a minority of about 11% of models (21% at the lower bound; post hoc arithmetic). Inflation that does not fade within 90 days, or that established models carry as well, is not measured; by day 90 a median 82% of a model's votes had been added after first listing (post hoc check). The mean is not robustly below zero: with five other seeds the upper bound runs from -0.004 to +0.008, and the bounds are reproducible only to about plus or minus 0.03 (Monte Carlo SD 0.016 over 200 seeds, post hoc check), and the bootstrap clustered by organization gives -2.68 to 0.75 (clustered two ways, by organization and listing date, -3.40 to 1.39, post hoc check). Every registered subgroup and sensitivity interval lies inside the margin; they are smaller subsets of the same data, not independent confirmation, and include the subset without style control (n = 344, +0.24, -0.87 to 1.42). The provider contrast (Meta plus Google minus all others, -0.75, -2.95 to 1.44) is exploratory. One stop rule depends on a definition fixed after the plan was committed but before any outcome was computed; under the other reading, 21.4% of models would have a reference panel under 30 models, above the 20% limit, and the study would have stopped. A second AI session reproduced the cohort and the primary point estimates. The results say nothing about any single model and nothing about cause. The record holds the note (PDF and Markdown), the frozen plan and the gate files (internal references replaced), the deviations log, the analysis result summary, the independent recheck with its scripts, the crosswalk, cohort and per-model tables, the outcomes file, the analysis code, the post hoc checks asked for by the reviews (post_review_checks.py, post_review_checks.json, post_review_checks_r2.py, post_review_checks_r2.json), and a trace of every number in the note. The source dataset is not redistributed; it is cited by revision. Derived tables are adapted from lmarena-ai/leaderboard-dataset by LMArena under CC BY 4.0. Plan, code, analysis and text by Claude (an AI model by Anthropic) for the author; the recheck by a second Claude session; no human has checked the figures; the human author is responsible for the deposit.
Authors
- Keisuke Watanabe (ORCID: https://orcid.org/0000-0001-6374-3918)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24367929
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00