Using ChatGPT as an AWE tool: quality, precision, and accuracy of the feedback

The emergence of generative pretrained transformer (GPT) large language models (LLMs) like ChatGPT has prompted speculation about their potential to serve as reliable Automated writing evaluation (AWE) tools and provide corrective feedback to second language (L2) writers. Given the novelty of this tool, research on this topic is scarce. Therefore, it is imperative to assess its appropriateness as an AWE tool before its implementation in real learning settings. To help fill this research gap, the current study employs both quantitative and qualitative methods to evaluate the quality, precision, and accuracy of the feedback generated by a customized GPT functioning as an AWE tool for 30 compositions in English. The results indicate that while the general accuracy rate is high and the tool can provide feedback on both form and content, there are occasional instances of erroneous feedback or fabricated mistakes. The educational implications of these findings are discussed.