Evaluating ChatGPT’s Efficacy: Automated vs. Human Scoring of EFL Students’ Summary Writing

This study examines ChatGPT’s potential as an assessment tool by comparing its performance with human raters in English as a Foreign Language (EFL) writing assessment conducted in a university classroom. Using a mixed-methods approach, 14 writing samples from university students in a Korean teacher education program were assessed through analytical and holistic scoring by ChatGPT and two human raters. Quantitative analyses measured intra- and inter-rater reliability as well as scoring leniency, while qualitative analyses explored differences in descriptive feedback. The findings indicate that ChatGPT demonstrates strong consistency in analytical scoring but greater variability in holistic scoring, highlighting the need for clearly defined evaluation criteria. While ChatGPT provides balanced feedback by addressing both content accuracy and language use, its comments remain general and lack detailed grammatical corrections. These findings suggest that AI tools can complement writing assessment by providing initial, broad feedback but require human oversight for more specific and nuanced evaluations. The study highlights the importance of well-designed prompts and integrating AI with traditional assessment methods to enhance EFL writing instruction.