The impact of GenAI on applicant behaviour, performance, and interview reliability during virtual interviews for medical school admissions

As artificial intelligence becomes increasingly powerful and accessible, education programs must guard against risks during student selection. Chief among those is avoidance of rewarding applicants who use prohibited tools. The trend towards increased use of virtual interviews makes programs particularly susceptible. To empirically study the risk and mitigation strategies, we conducted a randomized experiment comparing preparatory behaviors, performance, reliability and acceptability for candidates encouraged to surreptitiously use ChatGPT relative to two control groups. No advantages were observed (ChatGPT group mean = 3.67 (sd = 0.69) vs 3.74 (sd = 0.61) for 'usual practice' controls and 3.73 (sd = 0.80) for participants who were foretold station content). Reducing the time interviewees had to engage with ChatGPT off camera did not harm performance (3.73 (sd = 0.58) vs 3.70 (sd = 0.81) for controls); precision of test scores (SEMeas = 0.34 vs 0.39); or acceptability ratings (mean = 4.0 vs 4.1). These findings suggest an easy way to increase confidence in the fairness of virtual interviews.