Enhancing Teaching and Learning with Conversational AI Agents: Pilot Testing and Impact Assessment in Real Classroom Settings

This paper presents the evaluation of the educational impact of two conversational AI applications, one designed to support student self-study (Study Buddy) and the other to assist teachers in instructional planning (Teacher Mate) within the context of the AI4EDU project. The evaluation was conducted during the second piloting phase across four European countries in real classroom settings. In Greece, the evaluation involved 148 participants, 3 teachers and 145 students. The evaluation protocol included training workshops, pre- and post-intervention surveys, system interaction data analysis, and classroom-based implementation. Greek participants were assigned to experimental and control groups for comparative analysis. During a multi-week period, students used Study Buddy during self-study both at school and at home, while teachers integrated Teacher Mate in lesson planning and student monitoring tasks. Quantitative data from motivation and engagement questionnaires and interaction logs were triangulated with qualitative data from semi-structured interviews and classroom observations. The analysis provided insights into the impact of AI tools on users’ educational performance, engagement and motivation and helped identify possible drawbacks and key aspects of AI implementation that can be exploited to demonstrate the issues of ethics, transparency and inclusive use of AI in real educational settings. In general, participants from both groups expressed favorable opinions of the AI-driven applications. The combination of our study’s empirical results emphasizes the need to align AI tools with user needs, deliver customized training, and cultivate trust and openness of AI applications. These initiatives are critical for fostering inclusive, stimulating, and impactful AI-supported educational settings.