Student Performance on the American Board of Orthodontics Written Examination Following Flipped Classroom and Generative AI Approach

The flipped classroom approach enhances student learning by increasing engagement, reducing lecture time, and fostering collaboration [1]. Generative Artificial Intelligence (GenAI) tools can further enrich this method by efficiently generating educational content and offering customized feedback. Effectiveness of GenAI has shown students performed better academically and assisted medical students in learning [2, 3]. However, large language models (LLMs) can produce inaccuracies, outdated content, lack domain-specific expertise, and offer few source references, complicating information verification [4]. The University of Michigan recently introduced “Maizey”, a Retrieval-augmented Generation (RAG) framework within a LLM. RAG grounds LLM outputs in external, verifiable facts, enhancing response accuracy and reliability [5]. Maizey allows faculty to create and share with students a custom GenAI GPT tool such as an “Artificial Intelligence Teaching Assistant (AITA)” that uses specific datasets within a learning management system (LMS). This preliminary study evaluated 2nd year orthodontic residents’ performance on the national American Board of Orthodontics (ABO) Written Examination following the implementation of the flipped classroom methodology paired with AITA. Using the Maizey tool, AITA was incorporated with the LMS course site (CANVAS), which was used with previous cohorts and had up-to-date content, including the latest ABO reading list, textbook and lecture summaries, journal articles, and slide presentations. Similar to prior years, the students received approximately 850 multiple-choice questions (MCQs) based on the ABO examination content for 3 of the 4 modules; Basic and Applied Biomedical Sciences, Clinical Sciences A and B [6]. Module 4 material, Clinical Case Analysis, was not included in this course. Residents equally divided and answered MCQs using AITA prior to class. The residents cited references and supplemented with additional resources when they deemed it necessary. Classroom activities spanned 2-hour weekly sessions over 10 weeks, focusing on reviewing answers, discussing findings, and conducting further research. Performance on the ABO Written Examination for residents who took the course in the 3 previous years using the same course content but with faculty-expert lectures (Group A, n = 21) were compared with residents who used the flipped classroom and AITA method (Group B, n = 7) using T-test (p<0.05). There was significant improvement in overall examination performance for modules 1, 2, and 3 for Group B when compared to Group A (p≤0.001) (Table 1). Group B performed in the highest quintiles (1 = top 20% and 2 = above average) when compared with Group A and the national average scores. There was no difference in performance between the groups for module 4, which was not included in this course. Module 1: Basic and Applied Biomedical Sciences Mean (SD) Module 2: Clinical Sciences “A” Mean (SD) Module 3: Clinical Sciences “B” Mean (SD) Module 4: Clinical Case Analysis Mean (SD) Total Score Mean (SD) Group A (n = 21) Group B (n = 7) Significance (p<0.05)* The barrier to implementing flipped classroom approach is that faculty find sourcing materials for a flipped classroom challenging, and students may resent the increased out-of-class workload [3]. However, using the LMS for sourcing materials minimized the implementation time of the flipped classroom setup. The major benefit of this method was that references and sources for AITA's response were provided, allowing the student to verify the information. Residents reported high satisfaction with AITA, emphasizing its efficient study support, which increased their understanding and helped identify knowledge gaps. Although increasing the sample size, outside–of–classroom study activities and other potential confounding variables needs further investigation, the preliminary results suggest that incorporating flipped classroom with GenAI tools like AITA can significantly improve pedagogy and learning experiences.