Usability and Perceived Realism of an Artificial Intelligence Assisted Acute Care Simulator for Medical Students: Prospective Survey and Qualitative Evaluation (Preprint)

BACKGROUND Artificial intelligence (AI) assisted simulation can combine patient dialogue with changing clinical observations. Its usefulness in medical education depends on how learners experience the whole system, including the interface, scenario behavior, and teaching support. OBJECTIVE To evaluate the usability, acceptability, and perceived realism of BristolSimLab during supervised acute care teaching, and identify design features and limitations that could inform further development. METHODS We evaluated supervised use by 42 third-year medical students during a prehospital emergency medicine field trip. Students completed two acute care scenarios in pairs, with AI-generated feedback and support from clinical teaching fellows. Pre-session (n=40) and post-session (n=42) records assessed confidence before and after use, and usability, realism, and acceptability afterward. Free-text responses underwent descriptive content analysis. A concurrent incident log recorded detected system issues. RESULTS The mean System Usability Scale score was 81.3 (SD 10.4), and 41/42 respondents (97.6%) considered the activity acceptable. Clinically realistic responses to interventions were endorsed by 39/42 (92.9%), while 28/42 (66.7%) endorsed resemblance to managing a real acutely ill patient. Free-text categories included responsiveness or progression (16/42), temporal features (12/42), absent visual cues (12/42), and delegation through the nurse role (7/42). Confidence ratings were higher after use at the cohort level (P<.001). Mean scores on the different knowledge forms were both 6.5/8, which could not establish knowledge change. CONCLUSIONS BristolSimLab received favorable usability and acceptability ratings in supervised teaching. Learner feedback identified responsive scenario behavior as a strength and visual cues, role interaction, and feedback explanation as priorities for refinement. These findings support further task-based evaluation; they do not establish individual learning gains or a general system safety profile.