When agreement is not the goal: Reconsidering fairness in peer review

Critical peer review is the imperfect process by which knowledge building occurs in academia in the form of abstracts, manuscripts, and grant submissions. Peer review scrutinizes claims, exposes errors, and enables the gradual, collective correction of the scholarly record. This process has relied on human judgment since the Scientific Revolution; given the subjectivity inherent in human cognition, inter-rater reliability (IRR) has been identified as a challenge to fairness. Stephens et al explored this challenge by conducting a cross-sectional analysis of 1630 clinical vignette abstracts submitted to a large national hospitalist conference over two consecutive years.1 Each abstract was independently scored by three reviewers on a 1–10 scale along with a binary accept/reject recommendation. Consistent with prior literature, the authors found substantial inter-rater variation. Though among reviewers who participated in both years, scoring tendencies remained consistent over time. When scores were adjusted to account for reviewers' strictness or leniency, 11.2% of abstracts would have had their accept/reject outcome reversed. In this editorial, we explore two questions: (1) Is reproducible fairness the most important standard for peer review? (2) Is reproducible fairness achievable? Reproducible fairness is often invoked as a goal of scholarly evaluation, emphasizing consistency and transparency. However, consistency alone does not guarantee fairness. Even the opportunity to identify a case appropriate for submission as a clinical vignette is unfair, as it is subject to the randomness of clinical encounters. IRR metrics assume that agreement is the ideal outcome, yet in domains such as clinical vignettes, which both lack methodology and rely on contextual interpretation, some degree of disagreement is not only expected but appropriate. Tasks involving interpretation and judgment often yield lower reliability, not because raters are “incorrect,” but because the construct being evaluated is inherently subjective. Disagreement may therefore reflect diverse rater expertise and interests rather than flawed evaluation. Rather than pursuing the impossible task of removing rater subjectivity, incorporating mechanisms to understand rater cognition can help distinguish productive interpretive variation from undesired scoring noise. Incorporating qualitative reviewer feedback could provide a richer understanding of how reviewers interpret vignette quality and whether disagreement reflects the former or the latter. Narrative feedback also helps transform peer review from simply a ranking exercise into an educational process. Even if reproducible fairness should be the goal of peer review, the question remains how—and if—this can be achieved. The authors suggest several potential improvement efforts which we consider here, including enhanced reviewer orientation, structured rubrics, reviewer calibration session, score adjustment based on reviewer tendencies, and the use of artificial intelligence. In other domains of peer review, these interventions have demonstrated, at best, a modest improvement in IRR with significant persistent subjectivity. In one study, structured rubrics across 23 journals increased agreement only modestly (31%–41%).2 Rater training shows comparable limitations, with systematic reviews finding only weak evidence of marginal gains in peer review reliability.3 We have little reason to believe that these interventions would be any more effective in the setting of clinical vignettes, which lack methodology and standardization. Score adjustment based on rater tendencies has not been robustly explored with regard to peer review. However, it has been used with rater-based learner assessment.4 While this may provide some context for ratings, any modeling used for score adjustment comes with several assumptions that may vary by context and score use. Large language models (LLMs) can assist with pre-screening for structure, clarity, and topical fit, reducing administrative burden and reviewer fatigue. However, LLMs lack the contextual judgment needed to assess novelty, rigor, or equity-relevant contributions and should not be used for accept/reject decisions. When limited to submission pipeline tasks such as categorization and reviewer assignment and paired with human oversight, LLMs can potentially reduce avoidable process noise while preserving the meritocratic intent of peer review.5 Stephens et al add to a large body of literature demonstrating a notable lack of agreement across peer review decisions. These findings and their persistence across academic settings, even with structured interventions, call into question the heretofore unassailable goals of fairness and reproducibility. We posit that rater-based judgments are inherently subjective, and perhaps that fact is not an evil to be overcome, but a nuance to be navigated. No AI was used in the development of this manuscript. The authors declare no conflicts of interest.