Examining AI-Based Accuracy Assessment in L2 Learners’ Writing

Despite the increasing interest in using AI tools as assistant agents in instructional settings, the effectiveness of ChatGPT, the generative pretrained AI, for evaluating the accuracy of second language (L2) writing has been largely unexplored in formative assessment. Therefore, the current study aims to examine how ChatGPT, as an evaluator, performs in evaluating the accuracy of L2 writing, compared to human evaluators. Thirty-one academic writings were assessed by two human raters and ChatGPT. Both human and AI raters evaluated L2 learners’writings based on holistic measures (i.e., IELTS academic writing rubrics), focusing on coherence, lexical resources, grammatical accuracy, and global and specific measures of accuracy, by counting grammatical and lexical errors. The results revealed a moderate positive correlation in scores from holistic measures by humans and ChatGPT. This finding suggests that AI tools have the potential to assist both human raters and EFL learners in evaluating L2 writing, thereby highlighting the effective potential of employing AI tools as pedagogical aids within educational settings.