Does USMLE Performance Predict Residency Milestone Ratings in Medical Knowledge and Patient Care? A Multi-Site, Multi-Specialty Study

PURPOSE: Medical knowledge is a critical competency for physicians. Assessment of medical knowledge is distinguished between acquisition and application. Studies have demonstrated the association in acquisition throughout training; however, few have assessed the association between acquisition and application to practice. The purpose of this study was to explore the relationship between medical knowledge acquisition and application across a nationally representative data set. METHOD: This was a multi-institutional, multi-specialty retrospective study of data from medical school graduates at seven institutions entering residency training from 2016 to 2018. Medical knowledge acquisition was assessed using scores on United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge. Medical knowledge application was assessed using Accreditation Council for Graduate Medical Education (ACGME) milestone ratings during internship. ​​Mixed-effects regression was used to estimate the predictive association between USMLE performance and ACGME milestones, clustering specialty, and program effects using coefficients reflecting beta estimates. RESULTS: Data were analyzed from 3,430 medical school graduates. Failing USMLE Step 1 was associated with a higher risk of being rated as "not yet level 1" in patient care milestones at mid-year (OR = 4.19, P = .007) while failure of USMLE Step 2 was associated with the same in medical knowledge (OR = 7.56, P = .15). Higher USMLE scores were associated with higher milestone ratings, though the effect size was very small (standard coefficient range .01 - .19). There were no significant associations between USMLE scores and end-of-year milestone ratings for any analyses. CONCLUSIONS: Previous data suggest USMLE scores provide value in identifying future standardized examination (ie, medical knowledge acquisition) performance. These findings demonstrate less compelling evidence using USMLE to predict application of knowledge into practice. USMLE pass/fail status is the most predictive metric of medical knowledge application and may be useful in creating targeted remediation plans.