Impact of an Adaptive AI Chatbot for Formative Assessment: Experimental Study on Learning Dynamics and Medical Language Competence Among Nursing Students. (Preprint)

<sec> <title>BACKGROUND</title> Artificial intelligence (AI) is increasingly used in education to create personalized learning experiences. In nursing education, mastering medical language is essential but challenging, and traditional formative assessments often do not adapt to individual learner needs. Adaptive AI chatbots that tailor questions and feedback based on learner progress show promise for improving engagement and learning outcomes. </sec> <sec> <title>OBJECTIVE</title> This study examines the impact of the adaptive AI chatbot IA-Terma on nursing students’ academic performance, motivation, and engagement during formative assessments, with a focus on medical language acquisition. </sec> <sec> <title>METHODS</title> A randomized crossover design involved 28 nursing students divided into two groups. In the first phase, Group A used IA-Terma for self-assessment, while Group B experienced traditional assessments; roles switched in the second phase. IA-Terma’s adaptive system relies on a detailed mapping of medical language questions organized by difficulty and topic. Based on each student’s responses, the chatbot dynamically adjusts the difficulty of subsequent questions and provides personalized feedback tailored to individual progress. This approach ensures that learners are continually challenged at an appropriate level and receive targeted guidance. Academic skills were measured through written and oral tests, while motivation and cognitive engagement were assessed via questionnaires. Statistical analyses included paired t-tests, correlations, regressions, and mediation analysis to explore relationships among chatbot use, achievement, and engagement. </sec> <sec> <title>RESULTS</title> The use of the IA-Terma chatbot significantly influenced students’ academic performance, motivation, and cognitive engagement. For knowledge, Group A achieved higher scores than the control group B in Period 1 during chatbot use (t = 9.64, P &lt; .001), and the intragroup comparison revealed a decrease in scores following its withdrawal in Period 2 (t = 6.44, P &lt; .001). Comparable effects were observed for writing and pronunciation (P &lt; .001). Regarding motivation and engagement, the intragroup comparison indicated higher levels during the chatbot usage period (motivation: t = 9.90, P &lt; .001; engagement: t = 8.29, P &lt; .001), whereas the intergroup comparison showed higher scores for Group A in Period 1 (motivation: t = 6.07, P&lt; .001; engagement: t = 7.65, P &lt; .001) and a relative decline after the chatbot was withdrawn in Period 2 (motivation: t = -8.13, P &lt; .001; engagement: t = -10.2, P&lt; 0.001). The mediation analysis showed that the effect of motivation on performance was both direct (β = 0.750, p &lt; 0.001) and indirect through perseverance and cognitive engagement (β = 0.508, p = 0.002, 95% CI [0.0942, 0.5581]), highlighting the central role of these mediators in improving scores. </sec> <sec> <title>CONCLUSIONS</title> Integrating the adaptive AI chatbot IA-Terma into formative assessments enhances both academic outcomes and student engagement in nursing education. The dynamic adjustment of question difficulty and personalized feedback support more effective and sustained learning. These findings advocate for the broader adoption of adaptive AI tools in health sciences education to better address diverse learner needs and optimize educational results. </sec>