Shifting Verdicts, Persistent Ranking Signals: Twenty Frontier LLMs Review ICLR 2027 Submissions
Studies of large language model (LLM) reviewers mostly use papers whose decisions are already public, and treat a model as a fixed instrument. We test both assumptions on ICLR 2027 submissions before their decisions. On 8-14 November 2026, twenty models from OpenAI, Google and Anthropic released in 2026 reviewed the same stratified sample of 24 submissions with the official review form and web access disabled, and four of them reviewed 119 more. All twenty models rate nearly every paper a 2 ("weak reject") or a 3 ("weak accept"), with 2 of their 475 reviews at an extreme rating against 17-20% of human reviews, and two GPT models gave a 1 or 2 in 34 of their 40 reviews of unrecognized award papers. Against the same two-human reference, the median model's percentiles reach a Spearman correlation of 0.37 and a single human reviewer 0.36. Which middle rating dominates drifts across versions of a product line. On identical papers and prompts, the share placed on the accept side fell from 82% for Claude Opus 4.6 to 12% for Opus 5.5 (paired difference -68 points, 95% CI [-86, -50]), with a -44 point drop on 57 further papers, and from 79% for Gemini 3.8 Flash to 8% for Gemini 4 Hydrogen. Re-runs move 1 to 5 papers across the threshold in both directions, version changes 15 to 17, all toward rejection. Opus versions order papers similarly (median Spearman 0.77). From Opus 4.7 on, Claude models raise novelty in 46-83% of reviews, against 0-21% for GPT and Gemini models. In a title-only test, Claude Opus 5.5 recognized 28 of 60 decided ICLR 2026 papers, named the true first author in all 27 attempts and recognized none of 20 invented titles. We register a prospective test against the ICLR 2027 decisions. An LLM reviewer's absolute score is a version-specific quantity. Preprint; not peer reviewed. 11 pages, 4 figures, 1 table.
Authors
- Jizhou Guo
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24220599
- Primary Topic
- Ethics and Social Impacts of AI
- Type
- preprint