ChatGPT’s impact on student learning outcomes: a meta-analysis of 35 experimental studies

With the rapid advancement of generative artificial intelligence (GenAI) technology, the potential educational applications of Chat generative pre-trained transformers (ChatGPT) have attracted significant attention. However, the research on the specific effects of ChatGPT on student learning outcomes and its moderating factors remains insufficient. This study aimed to quantify the effects of ChatGPT on student learning outcomes and explore relevant moderating variables using a meta-analysis approach. The analysis included 35 studies published between 2022 and 2024, involving 4193 participants. The results indicated a moderately positive effect of ChatGPT on student learning outcomes (g = 0.670), significantly enhancing both cognitive and non-cognitive skills. In the analysis of moderating variables, the subject, experimental duration, and instructional mode had significant positive effects on student learning outcomes, whereas educational level and knowledge type did not show significant effects. Additionally, the publication bias test revealed no significant publication bias. This meta-analysis confirmed the effectiveness of ChatGPT in improving student learning outcomes and highlighted the roles of the subjects, experimental duration, and instructional mode as key moderating factors. Despite the risks of sample selection bias and limitations in fully covering the multidimensional moderating factors and higher-order thinking, the findings provided important empirical support for applying ChatGPT in education. Future research should investigate the mechanisms behind the impact of ChatGPT and consider a wider range of moderating factors to thoroughly evaluate its long-term effects on student learning outcomes.