Variation and inter‐rater reliability of abstract reviewer scores for clinical vignettes submitted to a national hospitalist meeting

Peer review of research products suffers from poor inter-rater reliability. Few studies examine whether this limitation generalizes to case reports. We conducted a cross-sectional analysis of peer reviews of clinical vignette abstracts submitted to a national hospitalist meeting in 2024 and 2025. Three randomly assigned reviewers scored each vignette on a 1-10 scale. We analyzed variation in scores across abstracts and reviewers and estimated inter-rater reliability via intraclass correlation coefficient (ICC). Two hundred twenty-one reviewers evaluated 1630 abstracts in 2024-2025. Abstract scores varied substantially: 384/1630 (23.6%) abstracts had a difference of 4 or more points (>2 standard deviations) between highest and lowest reviewer scores. Scores varied by reviewer: 2024 reviewer-level mean scores ranged 4.27-8.47 (standard deviation (SD): 0.70-2.80); 2025 scores ranged 4.06-8.59 (SD: 0.62-2.69). Inter-rater reliability was poor (ICC: 0.37). Adjusting final scores based on reviewer scoring tendencies changed the accept/reject category for 183 (11.2%) abstracts, suggesting opportunities for quality improvement.