Association Between AI-Associated Writing Characteristics in Program Applications and Application Outcomes: A Retrospective Observational Study

Abstract Background Use of artificial intelligence (AI) tools in program applications is increasing, yet their impact on application review and selection remains unclear. We examined the association between AI use, reviewer scores, and selection outcomes in a competitive emergency medicine undergraduate research associate program. Methods We conducted an observational study of applications submitted to an emergency medicine undergraduate research associate program from 2022–2025. Applications consisted of 4 questions (<500 words per response) and were scored by 2 independent reviewers (range 0-100). We estimated AI use on responses using a commercial AI-detection software (Winston AI) with higher scores (range 0-100) indicating greater likelihood of AI use. Outcomes included overall application score, interview invitation, and program acceptance. We conducted analyses evaluating the likelihood of AI use in predicting outcomes adjusting for application year, quarter, gender, race/ethnicity, class year, and word count. We also conducted a sensitivity analysis evaluating the association between likelihood of AI use and acceptance restricted to applicants who received interviews. Results Among 1,801 applications, 30% received interview invitations and 8% were accepted. Likelihood of AI use increased each year. Higher likelihood of AI use was independently associated with higher scores (p<0.001). Applications with the highest likelihood of AI use (90-100%) were also associated with interview invitations (adjusted odds ratio [aOR] 4.1, 95% CI 2.9–5.7) and acceptance (aOR 1.8, 95% CI 1.0-3.0). When restricted to applicants receiving interviews, likelihood of AI use was not independently associated with acceptance. Conclusion Likelihood of AI use in program applications increased each year and was associated with higher reviewer scores and interview invitations but was not independently associated with program acceptance among interviewed applicants. These findings highlight emerging challenges in evaluating written application materials and the need for clearer guidance regarding AI use in program applications.