Frontier language models recover scientific quality but predictably diverge from human reviewers

Abstract Frontier language models are now deployed for peer review at scale, yet whether they recover established scientific quality hierarchies and how they diverge from human reviewers remain unclear. We evaluate 1,066 machine learning papers from 2025 main artificial intelligence conferences (NeurIPS, ICML, ICLR), matched workshops, and citation-stratified journals using three frontier models (GPT~5.4, Gemini~3.1~Pro, Claude~Opus~4.6) in consensus on a 4-point reviewing rubric. Consensus scores recover every expected quality signal: main-conference papers outscore workshops by 0.40 units ($p = 3.93 \times 10^{-55}$), presentation rankings emerge without access to acceptance metadata, and highly cited journal papers substantially outperform lowly cited counterparts. However, all three models converge on suppressing originality by 14--17 percentage points, and score variation across research areas reflects cross-model disagreement patterns. In a nested analysis of 162 ICLR papers, model-human alignment differs sharply between paper strengths (high agreement) versus weaknesses (low agreement). While frontier models agree with one another more than human reviewers agree with each other, strength-weakness alignment is only weakly coupled ($r \approx 0.17$--$0.19$), indicating this asymmetry is structural to review itself rather than an artifact of AI deployment. These findings establish that multi-model consensus recovers meaningful quality signals, but deployments require explicit accounting for originality underweighting and persistent gaps in weakness detection.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11132861/v1
Primary Topic
scientometrics and bibliometrics research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Frontier language models recover scientific quality but predictably diverge from human reviewers

Zhisheng Tang, Mayank Kejriwal, Vaibhava Lakshmi Ravideshik
Research Square
scientometrics and bibliometrics research
preprint

Frontier language models recover scientific quality but predictably diverge from human reviewers

Zhisheng Tang, Mayank Kejriwal, Vaibhava Lakshmi Ravideshik
preprint en

Abstract

Abstract Frontier language models are now deployed for peer review at scale, yet whether they recover established scientific quality hierarchies and how they diverge from human reviewers remain unclear. We evaluate 1,066 machine learning papers from 2025 main artificial intelligence conferences (NeurIPS, ICML, ICLR), matched workshops, and citation-stratified journals using three frontier models (GPT~5.4, Gemini~3.1~Pro, Claude~Opus~4.6) in consensus on a 4-point reviewing rubric. Consensus scores recover every expected quality signal: main-conference papers outscore workshops by 0.40 units ($p = 3.93 \times 10^{-55}$), presentation rankings emerge without access to acceptance metadata, and highly cited journal papers substantially outperform lowly cited counterparts. However, all three models converge on suppressing originality by 14--17 percentage points, and score variation across research areas reflects cross-model disagreement patterns. In a nested analysis of 162 ICLR papers, model-human alignment differs sharply between paper strengths (high agreement) versus weaknesses (low agreement). While frontier models agree with one another more than human reviewers agree with each other, strength-weakness alignment is only weakly coupled ($r \approx 0.17$--$0.19$), indicating this asymmetry is structural to review itself rather than an artifact of AI deployment. These findings establish that multi-model consensus recovers meaningful quality signals, but deployments require explicit accounting for originality underweighting and persistent gaps in weakness detection.

Research Square
University of Southern California (US)
scientometrics and bibliometrics research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon