Do competency-aligned admissions assessments predict medical school awards? A validity argument using Kane’s framework

Abstract Background Medical school admissions committees strive to use evidence-based, valid measures to select applicants who will succeed in training and excel in medical practice. Guided by Kane’s Validity Framework, this study builds a validity argument by examining predictive validity for four competency-based admissions assessments (GPA, MCAT, Casper SJT, and local MMI) in relation to academic and professional awards conferred during medical school training. Methods A retrospective study was conducted on 644 graduates (2020–2022) from New York Medical College School of Medicine (NYMC SOM). Stepwise binomial logistic regression models assessed the association of the admissions assessments independently in predicting award attainment. Chi-square tests and contingency tables were used to examine the concordance between high admissions scores and award recipients. Results MCAT scores were significantly associated with model improvement for Academic awards (OR = 1.43; 95% CI, 1.13–1.80, p < .001). Casper SJT and MMI scores were significantly associated with Service awards model improvement (OR = 1.24; 95% CI, 1.00-1.55, p = .010; OR = 1.39; 95% CI, 1.13–1.71, p = .002, respectively). Casper SJT scores were significantly associated with Clinical awards model improvement (OR, 1.46; 95% CI, 1.11–1.92, p < .003). Other metrics were non-significant. Conclusions This study applies Kane’s framework to examine associations between admissions assessments and institutionally recognized award outcomes across academic and professional domains. The findings provide context-specific validity evidence relevant to the extrapolation inference and illustrate the potential value of considering broader outcome measures beyond traditional academic performance. Kane’s framework may be useful for institutions seeking to evaluate admissions assessments in relation to outcomes aligned with their educational goals.