The potential advantages of using an LLM-based chatbot for automated writing evaluation for English teaching practitioners

With the ever-increasing demand for assessing large amounts of student writing, Automated Writing Evaluation (AWE) has emerged as an efficient system to satisfy this demand. However, it has also been suggested that applying it in teachers’ writing classes may offer limited results. Given this, the present study developed an AWE chatbot based on ChatGPT 4.0-turbo, designed as an automated rater. A total of 465 narrative essays written by Korean high school EFL students were scored according to three criteria by the developed tool; these were then compared with the scores administered by two professional raters using various analytic measures. The results showed that the AWE chatbot’s scoring was strongly correlated with that of the human raters. Meanwhile, the many-facet Rasch model’s result indicated that the two human raters’ statistics demonstrated an excellent fit, whereas those of the developed AWE chatbot were slightly lower. The Coh-Metrix analysis suggested that the human raters’ scoring tendencies and GPT are largely aligned, indicating that both raters scored the essays similarly. Based on our findings, we suggest that the Large Language Model (LLM)-based AWE chatbot has great potential to assist teachers in EFL writing classrooms.