Assessing the reliability and relevance of DeepSeek in EFL writing evaluation: a generalizability theory approach

This study explored the potential of DeepSeek to contribute to English writing assessment via generalizability (G-) theory and qualitative feedback analysis. Specifically, it assessed the reliability of holistic scores and qualitative feedback generated by DeepSeek-V3 and DeepSeek-R1 for essays written in English as foreign language (EFL) learners, and compared these scores with those assigned by four college English teachers. The data consisted of 92 College English Test Band 4 (CET-4) essays written by non-English majors at a university in Heilongjiang Province, China. All the essays were holistically scored by DeepSeek-V3, R1, and four college English teachers. In addition, all three groups of raters provided qualitative feedback on content, language use, organization, and coherence. G-theory analysis revealed that the scoring reliability of DeepSeek-V3 was consistently lower than that of DeepSeek-R1 and the teacher raters; however, DeepSeek-R1 demonstrated consistently higher reliability coefficients than the teachers did. The qualitative feedback analysis indicated that both DeepSeek-V3 and R1 consistently provided more relevant feedback on the EFL essays than did the teacher raters. Furthermore, DeepSeek-V3 and R1 were equally relevant across the content, language use, organization, and coherence aspects of the essays, whereas the teacher raters generally focused more on language use but provided less relevant feedback on content, organization, and coherence. Consequently, DeepSeek-V3 and R1 could be useful AI tools for enhancing EFL writing assessments. The implications of adopting DeepSeek for classroom writing assessments are discussed.