Chatbots in Education: Hype or Help? A Meta-Analysis

Artificial intelligence integration in education, primarily through chatbots, has emerged as a potential solution to address the challenges of catering to students’ diverse learning backgrounds. This meta-analysis examined chatbot effectiveness in education, driven by amplified interest since ChatGPT’s introduction in 2022. Adhering to PRISMA guidelines, we analyzed 135 effect sizes from 62 studies across diverse educational contexts. Initial results revealed a large positive effect of chatbots on learning performance. However, after controlling for publication bias, we found a significant small to moderate effect. Interaction mode, field of study, intervention duration, and educational level emerged as significant moderators. Text-based interactions, STEM disciplines, longer interventions, and lower educational levels showed the most substantial chatbot effectiveness in enhancing learning performance. Neither chatbot type nor cultural differences significantly influenced learning performance. While our findings support chatbots’ potential to enhance performance, they underscore the necessity for targeted implementation when integrating these technologies into learning environments. Keywords: artificial intelligence, chatbots, education, meta-analysis