Chinese Generative AI in HSK Reading Comprehension: An In-depth Analysis of ERNIE Bot 4.0

This study evaluated the Chinese reading comprehension abilities of ERNIE Bot, a leading Chinese generative AI, using HSK reading test items. Results showed that ERNIE Bot demonstrates excellent overall Chinese reading comprehension skills, with the pre-trained model excelling in vocabulary and discourse domains, while the fine-tuned model showed significant improvements in grammar and cohesion areas. ERNIE Bot performed best with longer texts providing ample context, and its recently expanded prompt length capacity allows for inputting comprehensive HSK test items across various domains.<br/> The research’s significance lies in its systematic analysis of a Chinese AI’s reading comprehension abilities, suggesting potential applications in Chinese language education and assessment. By comparing pre-trained and fine-tuned models, the study verified ERNIE Bot’s learning capacity and improvement potential.<br/> However, ERNIE Bot showed limitations in understanding basic test-taking rules, logical relationships between short sentences, and comprehension of isolated sentences. Overcoming these limitations requires careful prompt design and involvement of education experts.<br/> Future research should focus on comparative studies of various Chinese AI models, practical application studies in educational settings, and methodological research to address ERNIE Bot’s limitations. In conclusion, ERNIE Bot demonstrated high-level performance in Chinese reading comprehension, with potential for improvement through fine-tuning, promising qualitative advancements in Chinese language education through further research and practical applications.