ChatGPT4o as an AI Peer Assessor in EFL Speaking Classrooms: Examining Scoring Reliability and Feedback Effectiveness

This study investigated the role of ChatGPT4o as an AI peer assessor in English-as-a-foreign-language (EFL) speaking classrooms, with a focus on its scoring reliability and the effectiveness of its feedback. The research involved 40 first-year English major students from two parallel classes at a Chinese university. Twenty from one class served as speech sample providers; and the other 20 served as human peer assessors. In addition, ChatGPT4o served as an AI peer assessor. The study employed univariate and multivariate generalizability (G-) theory to compare the consistency and reliability of holistic and analytic scoring between ChatGPT4o and human peer assessors. The results demonstrated that ChatGPT4o provided significantly more consistent and reliable scores across domains such as accuracy, fluency, and complexity. Moreover, ChatGPT4o delivered more comprehensive and effective feedback, offering clear guidance for improvement. However, interviews with human peer assessors revealed concerns about ChatGPT4o’s limitations in capturing the subtle aspects of spoken language, such as emotion and cultural context, and the potential over-reliance on ChatGPT4o in EFL assessments. The findings suggested that while ChatGPT4o as an AI peer assessor can enhance the reliability and quality of peer assessments, its adoption should be carefully managed to complement, rather than replace, human judgment, ensuring a balanced approach in EFL speaking classrooms.