Frontier language models recover scientific quality but predictably diverge from human reviewers
Abstract Frontier language models are now deployed for peer review at scale, yet whether they recover established scientific quality hierarchies and how they diverge from human reviewers remain unclear. We evaluate 1,066 machine learning papers from 2025 main artificial intelligence conferences (NeurIPS, ICML, ICLR), matched workshops, and citation-stratified journals using three frontier models (GPT~5.4, Gemini~3.1~Pro, Claude~Opus~4.6) in consensus on a 4-point reviewing rubric. Consensus scores recover every expected quality signal: main-conference papers outscore workshops by 0.40 units ($p = 3.93 \times 10^{-55}$), presentation rankings emerge without access to acceptance metadata, and highly cited journal papers substantially outperform lowly cited counterparts. However, all three models converge on suppressing originality by 14--17 percentage points, and score variation across research areas reflects cross-model disagreement patterns. In a nested analysis of 162 ICLR papers, model-human alignment differs sharply between paper strengths (high agreement) versus weaknesses (low agreement). While frontier models agree with one another more than human reviewers agree with each other, strength-weakness alignment is only weakly coupled ($r \approx 0.17$--$0.19$), indicating this asymmetry is structural to review itself rather than an artifact of AI deployment. These findings establish that multi-model consensus recovers meaningful quality signals, but deployments require explicit accounting for originality underweighting and persistent gaps in weakness detection.
Authors
- Zhisheng Tang (ORCID: https://orcid.org/0000-0002-7347-3701)
- Mayank Kejriwal (ORCID: https://orcid.org/0000-0001-5988-8305)
- Vaibhava Lakshmi Ravideshik
Institutions
- University of Southern California (US)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-12
- DOI
- https://doi.org/10.21203/rs.3.rs-11132861/v1
- Primary Topic
- scientometrics and bibliometrics research
- Type
- preprint