A Reliability Assessment of DeepSeek-V3.2 in Evaluating International Students’ Master’s Theses in International Chinese Education

This study investigates the inter-rater reliability of DeepSeek-V3.2 in evaluating international students’ degree theses by comparing its scoring results with those of human experts in International Chinese Education. The results indicate an overall moderate agreement between the artificial intelligence (AI) and human expert evaluations (ICC = 0.622, r = 0.641). Beneath this overall agreement, however, AI exhibited a marked scoring bias relative to human experts: it significantly overrated performance on “Logical Structure and Coherence” and “Accuracy of Disciplinary Knowledge,” while significantly underrating “Use of Literature,” “Academic Norms,” “Critical Thinking,” and “Innovation.” Its ratings were most closely aligned with those of human experts only on the dimension of “Academic Chinese Awareness.” These findings suggest that while DeepSeek-V3.2 is capable of identifying superficial, form-oriented linguistic issues in international students’ theses, it has significant limitations in evaluating complex cognitive dimensions such as deeper logic and innovative thinking. Notably, the AI’s inflated rating on the logic-structure dimension indicates that it responded to the surface presentation of logic rather than its substantive quality. Therefore, although DeepSeek-V3.2 can serve as a preliminary auxiliary tool in the human–machine collaboration process, it cannot replace the comprehensive judgment of human experts.