ChatGPT: A reliable assistant for the evaluation of students’ written texts?

Abstract There is growing interest in the potential of Artificial Intelligence (AI) to assist in various educational tasks, including writing assessment. However, the comparative efficacy of human and AI-powered systems in this domain remains a subject of ongoing exploration. This study aimed to compare the accuracy of human raters (teachers and pre-service teachers) and AI systems (ChatGPT and trained ChatGPT) in classifying written texts. The study employed both chi-square tests and logistic regression analysis to examine the relationship between rater groups (human vs. machine) and the accuracy of text classification. Initial chi-square analyses suggested no significant differences in classification accuracy between human and AI raters. However, the logistic regression model revealed a significant relationship, with human raters demonstrating a higher rate of correct classification compared to their AI counterparts. The logistic model achieved an 81.3% success rate in predicting correct classifications. While AI systems show promise in automated text processing, human raters currently demonstrate superior accuracy in writing assessment tasks. These findings highlight the need for further research into the strengths and limitations of both human and AI-based approaches. The integration of AI in educational assessment should focus on complementing and supporting, rather than replacing, the expertise of human educators.