Teacher vs Artificial Intelligence: a comparison of the quality of feedback provided by a teacher and generative artificial intelligence in assessing students' creative writing

Introduction. Writing foreign-language creative writing assignments is one of the goals of foreign language teaching in higher education. Modern AI tools (ChatGPT 4.0) are able to provide learners with evaluative feedback and recommendations for essay revision. However, the feedback quality from ChatGPT 4 and other AI tools is a subject of discussion in the teaching community. The aim of this paper is to compare the feedback quality provided by ChatGPT 4.0 and teachers in evaluating students' essays. Materials and methods. A bank of English essays (N=350) written by linguistics students (A2-B1 level) was used as the material. The participants of the research were 12 teachers of English at Derzhavin Tambov State University (Russian Federation). For every essay, one teacher and ChatGPT gave evaluative feedback on the following criteria: 1) content of the essay; 2) organisation and structure of the essay; 3) supporting ideas and arguments; 4) language of the essay (lexical aspect of speech, grammatical aspect of speech, syntax); and 5) originality of the idea. The recommendations received from the teacher and ChatGPT 4.0 were evaluated on the basis of norm-referenced testing. The data analysis was carried out using the Student’s t-test. Research results. It was found that ChatGPT matched the teacher in terms of the quality of evaluative feedback for the following criteria: ‘content of the paper’ (t=0.24; p>0.05), ‘organisation and structure’ (t=1; p>0.05), and ‘supporting ideas and arguments’ (t=1.43; p>0.05). Moreover, ChatGPT outperformed the teacher (not a native speaker) for the criteria ‘language of the essay’ (t=1.67; p≤0.05) and ‘originality of the essay’ (t=1.78; p≤0.05), which is explained by the fact that the GPT language model was developed based on large English textual data. This allowed the AI tool to be more accurate in assessing specifically the linguistic correctness of a written expression. Conclusion. The novelty of the study is in the confirmation of the ability of the AI tool ChatGPT 4.0 to provide qualitative feedback in assessing creative writing at the teacher's level or even better. The results of the study support more intensive implementation of ChatGPT 4.0 in the process of teaching foreign language and assessing the development level of students' writing skills.