Why Is Reviewer 2 Always the Villain? Myth, Bias, or Statistical Reality?

Abstract The Reviewer 2 stereotype conflates recommendation severity, polarization, disagreement, and editorial influence. We examined these dimensions using metadata-only records reconstructed from the Open Review-Based (ORB) archive, focusing on ordinal rating behavior rather than review-text tone or civility. After deduplication, the database contained 62,935 manuscripts and 157,070 review records. The primary explicit-label sample included 8,885 manuscripts and 29,166 reviews, concentrated predominantly in OpenReview machine-learning conferences (7,966 from ICLR 2018–2021). Manuscript-balanced ordinal contrasts, venue-specific mixed models, endpoint analyses, and independent editorial outcomes separated observable positions from decision mechanisms. Label two showed no clear excess harshness: -0.008 (95% CI -0.022 to 0.006). Its excess probability of either scale endpoint was 0.003 (95% CI -0.004 to 0.009); median-distance and broader-tail analyses likewise did not support general polarization. Public labels matched creation chronology on only 30.1% of eligible reviews. Among 5,964 decision manuscripts, greater second-reviewer severity was associated with rejection, but the severity-by-peer-disagreement interaction was inconclusive (P = 0.637). In a retrospective audit of 1,180 held-out final archived snapshots, an equal-weight mean outperformed a separate-R2 model in Brier loss (0.0830 vs. 0.0853). A paired MIDL subset showed reduced disagreement after rebuttal across 85 manuscripts, without clear second-reviewer-specific change. The supplied small F1000Research case was treated solely as an unverified historical audit, not an independent replication. Within these observed open-conference settings, the evidence favors a measurement- and institution-dependent account over universal Reviewer 2 ordinal harshness or polarization; generalizability to traditional cross-disciplinary journals, linguistic hostility, and prospective editorial weighting remain unmeasured.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11740871/v1
Primary Topic
Publishing and Scholarly Communication
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Why Is Reviewer 2 Always the Villain? Myth, Bias, or Statistical Reality?

Muh. Akbar Idris
1 links
Research Square
Publishing and Scholarly Communication
preprint

Why Is Reviewer 2 Always the Villain? Myth, Bias, or Statistical Reality?

Muh. Akbar Idris
preprint en

Abstract

Abstract The Reviewer 2 stereotype conflates recommendation severity, polarization, disagreement, and editorial influence. We examined these dimensions using metadata-only records reconstructed from the Open Review-Based (ORB) archive, focusing on ordinal rating behavior rather than review-text tone or civility. After deduplication, the database contained 62,935 manuscripts and 157,070 review records. The primary explicit-label sample included 8,885 manuscripts and 29,166 reviews, concentrated predominantly in OpenReview machine-learning conferences (7,966 from ICLR 2018–2021). Manuscript-balanced ordinal contrasts, venue-specific mixed models, endpoint analyses, and independent editorial outcomes separated observable positions from decision mechanisms. Label two showed no clear excess harshness: -0.008 (95% CI -0.022 to 0.006). Its excess probability of either scale endpoint was 0.003 (95% CI -0.004 to 0.009); median-distance and broader-tail analyses likewise did not support general polarization. Public labels matched creation chronology on only 30.1% of eligible reviews. Among 5,964 decision manuscripts, greater second-reviewer severity was associated with rejection, but the severity-by-peer-disagreement interaction was inconclusive (P = 0.637). In a retrospective audit of 1,180 held-out final archived snapshots, an equal-weight mean outperformed a separate-R2 model in Brier loss (0.0830 vs. 0.0853). A paired MIDL subset showed reduced disagreement after rebuttal across 85 manuscripts, without clear second-reviewer-specific change. The supplied small F1000Research case was treated solely as an unverified historical audit, not an independent replication. Within these observed open-conference settings, the evidence favors a measurement- and institution-dependent account over universal Reviewer 2 ordinal harshness or polarization; generalizability to traditional cross-disciplinary journals, linguistic hostility, and prospective editorial weighting remain unmeasured.

Research Square
IPB University (ID)
Publishing and Scholarly Communication
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon