Beyond grades to learning enhancement: development, implementation and evaluation of a conversational AI agent in medical education

Purpose The integration of conversational artificial intelligence (AI) in medical education shows promise but lacks empirical evidence of effectiveness. This study evaluated a customized generative pre-trained transformer (GPT)-4-based conversational agent designed explicitly for a pharmacology block in undergraduate medical education. Design/methodology/approach A mixed-methods sequential explanatory design was employed in a four-week elective pharmacology block at the College of Medicine, Sulaiman Al-Rajhi University. Phase 1 involved a cross-sectional survey of enrolled students (n = 58) who used the AI tool, with a comparative analysis of their academic performance against historical cohorts without access to AI (2022–2023: n = 33; 2023–2024: n = 37). Academic performance was measured through mid-block quizzes, end-of-block examinations, and total block score. Phase 2 comprised semi-structured interviews (n = 9), transcribed and analyzed thematically. Findings The survey response rate was 89.6% (52/58), and 98.1% (51/52) reported using the tool. Among users, 74.5% engaged with the GPT tool daily. Students reported exceptionally high satisfaction: 92.2% found it more effective than traditional methods, 94.1% felt it made learning more engaging, and the likelihood of recommending it averaged 4.65 ± 0.73 (on a 5-point scale). Frequency of use was significantly associated with perceived improvement in learning (χ2 = 15.84, p < 0.05). Academic performance showed marginal, non-significant improvements: total scores increased by 2.5% (75.52 vs. 77.40, p > 0.05), with pass rates improving from 92.9% to 98.3% (p > 0.05). Qualitative analysis revealed six themes: learning support and academic enhancement; efficiency and accessibility; GPT's supplementary role in learning; perceived strengths and unique features; limitations and challenges; and suggestions for improvement. Originality/value The disconnect between exceptional user engagement and minimal performance improvements challenges traditional evaluation paradigms. While conversational AI may not directly enhance examination scores, students' accounts suggest it may support metacognitive development, reduce cognitive load, and foster critical-thinking skills essential for modern medical practice, thereby necessitating multidimensional assessment frameworks.