Navigating Holistic Assessment in Pathology Residency Selection: A Single-Institution Evaluation of Domain Performance

Context.—: The shift from standardized metrics to holistic review in residency selection aims to capture a broader range of competencies essential for pathology practice. However, validation of such frameworks remains limited within specialty-specific contexts. Objective.—: To evaluate the measurement properties and structural integrity of a holistic assessment system used for pathology residency applicants. Design.—: Our program implemented holistic evaluation forms assessing 10 competency domains: General Interview, Communication Skills, Scholarly Activities, Recommendation Letters, Rigor of Education, Leadership, Serving the Underserved, Pathology Dedication, Interest in Program, and Adding a New Perspective. Faculty completed 280 assessments of 96 applicants across 2 National Resident Matching Program cycles (2023-2025). For each domain, we quantified discriminatory ability (variance normalized to scale range), interrater agreement using the intraclass correlation coefficient (ICC), and predictive importance for total applicant scores using random forest (RF) modeling. Results.—: Skill domains showed significant heterogeneity in both overall mean normalized scores (ANOVA [analysis of variance] F = 76.7; P < .001) and within-applicant scoring patterns (Friedman χ2 = 349; P < .001). Pathology Dedication showed the strongest performance, with the highest agreement (ICC = 0.54; rank 1) and high discrimination (spread = 0.09; rank 2). General Interview demonstrated minimal discrimination (spread = 0.02; rank 10) and lowest agreement (ICC = 0.08; rank 10). Adding a New Perspective and Recommendation Letters were the strongest predictors of overall scores (RF = 0.28 and 0.22; ranks 1-2), while Interest in Program-despite high discrimination-had minimal predictive value (RF = 0.02; rank 10). Conclusions.—: Specialty-specific domain (Pathology Dedication) provided reliable and predictive evaluations, whereas General Interview performance contributed least to consistent or meaningful applicant differentiation.