Why Is Reviewer 2 Always the Villain? Myth, Bias, or Statistical Reality?
Abstract The Reviewer 2 stereotype conflates recommendation severity, polarization, disagreement, and editorial influence. We examined these dimensions using metadata-only records reconstructed from the Open Review-Based (ORB) archive, focusing on ordinal rating behavior rather than review-text tone or civility. After deduplication, the database contained 62,935 manuscripts and 157,070 review records. The primary explicit-label sample included 8,885 manuscripts and 29,166 reviews, concentrated predominantly in OpenReview machine-learning conferences (7,966 from ICLR 2018–2021). Manuscript-balanced ordinal contrasts, venue-specific mixed models, endpoint analyses, and independent editorial outcomes separated observable positions from decision mechanisms. Label two showed no clear excess harshness: -0.008 (95% CI -0.022 to 0.006). Its excess probability of either scale endpoint was 0.003 (95% CI -0.004 to 0.009); median-distance and broader-tail analyses likewise did not support general polarization. Public labels matched creation chronology on only 30.1% of eligible reviews. Among 5,964 decision manuscripts, greater second-reviewer severity was associated with rejection, but the severity-by-peer-disagreement interaction was inconclusive (P = 0.637). In a retrospective audit of 1,180 held-out final archived snapshots, an equal-weight mean outperformed a separate-R2 model in Brier loss (0.0830 vs. 0.0853). A paired MIDL subset showed reduced disagreement after rebuttal across 85 manuscripts, without clear second-reviewer-specific change. The supplied small F1000Research case was treated solely as an unverified historical audit, not an independent replication. Within these observed open-conference settings, the evidence favors a measurement- and institution-dependent account over universal Reviewer 2 ordinal harshness or polarization; generalizability to traditional cross-disciplinary journals, linguistic hostility, and prospective editorial weighting remain unmeasured.
Authors
- Muh. Akbar Idris
Institutions
- IPB University (ID)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-17
- DOI
- https://doi.org/10.21203/rs.3.rs-11740871/v1
- Primary Topic
- Publishing and Scholarly Communication
- Type
- preprint