Assessing the Learning Gains and User Experience of Large Language Model Simulations for Clinical Medical Education: A Pilot Observational Study

Abstract Background Medical students often struggle transitioning from preclinical coursework to clinical environments. Simulation-based training can bridge this gap but can be resource intensive. Large language models (LLMs) provide a scalable resource that can augment traditional training. However, their effects on learning outcomes and student perceptions are currently not understood. This study aims to evaluate medical student knowledge acquisition and perception of the utility of simplified LLM-based clinical simulation. Methods Using the RISE framework, a structured simulation was designed to assess medical student knowledge of acute asthma exacerbation management. ZotGPT, an institutional GPT-4 based platform was utilized. Medical students completed pre- and post-simulation assessments assessing knowledge, confidence, and perceptions of simulation utility. Subgroup analysis was stratified by time spent completing the simulation. Statistical analyses included Wilcoxon signal-rank and McNemar tests. Results A total of 155 students participated. The simulation significantly improved perceived understanding of key treatment steps, including initial bronchodilator and steroid therapy (p = 0.001), magnesium sulfate escalation (p = 0.002), refractory management (p = 0.021), and discharge planning (p = 0.005). Students rated the simulation as more useful than non-simulation learning tools (p = 0.017) and noninferior to in-person simulation (p = 0.226). Confidence in asthma management increased (p < 0.001), but objective knowledge gains were not observed, with balanced correct-to-incorrect and incorrect-to-correct responses for intermediate steps. Longer engagement (> 600 seconds) was associated with greater perceived benefit. Simulations were deemed helpful for learning (94.7%), and most students preferred immediate feedback (68.4%). Conclusions LLM-based clinical simulations were well accepted, improved medical student confidence, and were perceived as educationally useful. However, objective knowledge gains were limited. LLMs thus may serve as an adjunctive educational tool but require improved prompt design, structured feedback, and guided implementation to optimize outcomes.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-10529565/v1
Primary Topic
Simulation-Based Education in Healthcare
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Assessing the Learning Gains and User Experience of Large Language Model Simulations for Clinical Medical Education: A Pilot Observational Study

Alisa Wray, Faris F Halaseh, Ronald J. Rivera, Saman Andalib et al.
8 links
Research Square
Simulation-Based Education in Healthcare
preprint

Assessing the Learning Gains and User Experience of Large Language Model Simulations for Clinical Medical Education: A Pilot Observational Study

Alisa Wray, Faris F Halaseh, Ronald J. Rivera, Saman Andalib, Soheil Saadat, Riley Scherr, Aidin Spina
preprint en

Abstract

Abstract Background Medical students often struggle transitioning from preclinical coursework to clinical environments. Simulation-based training can bridge this gap but can be resource intensive. Large language models (LLMs) provide a scalable resource that can augment traditional training. However, their effects on learning outcomes and student perceptions are currently not understood. This study aims to evaluate medical student knowledge acquisition and perception of the utility of simplified LLM-based clinical simulation. Methods Using the RISE framework, a structured simulation was designed to assess medical student knowledge of acute asthma exacerbation management. ZotGPT, an institutional GPT-4 based platform was utilized. Medical students completed pre- and post-simulation assessments assessing knowledge, confidence, and perceptions of simulation utility. Subgroup analysis was stratified by time spent completing the simulation. Statistical analyses included Wilcoxon signal-rank and McNemar tests. Results A total of 155 students participated. The simulation significantly improved perceived understanding of key treatment steps, including initial bronchodilator and steroid therapy (p = 0.001), magnesium sulfate escalation (p = 0.002), refractory management (p = 0.021), and discharge planning (p = 0.005). Students rated the simulation as more useful than non-simulation learning tools (p = 0.017) and noninferior to in-person simulation (p = 0.226). Confidence in asthma management increased (p < 0.001), but objective knowledge gains were not observed, with balanced correct-to-incorrect and incorrect-to-correct responses for intermediate steps. Longer engagement (> 600 seconds) was associated with greater perceived benefit. Simulations were deemed helpful for learning (94.7%), and most students preferred immediate feedback (68.4%). Conclusions LLM-based clinical simulations were well accepted, improved medical student confidence, and were perceived as educationally useful. However, objective knowledge gains were limited. LLMs thus may serve as an adjunctive educational tool but require improved prompt design, structured feedback, and guided implementation to optimize outcomes.

Research Square
Beth Israel Deaconess Medical Center (US), University of California, Irvine (US), Scripps Mercy Hospital (US), University of California, Irvine Medical Center (US)
Good health and well-being, Quality education
Simulation-Based Education in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon