Syntactic And Lexical Comparison Between Ai-generated Reading Passages And Japanese Universities' National Test

This research aimed to develop a methodology for creating mock English reading tests using AI-generated passages to help decrease the workload of EFL teachers who struggle in creating the high-quality mock English reading test for universities’ entrance examinations. To achieve this goal, this paper examined the lexical and syntactical differences between English Subject of the Center Test at Japanese universities’ entrance exams (ESCT) and AI-generated passages. Three different generative AI were used: OpenAI ChatGPT version 4, Google Gemini version 1.5 Flash, and DeepSeek-V3. To make the vocabulary coverage between AI-generated passages and ESCTs meaningful, topics based on ESCTs were used to create 11 prompts for AI-generated passages. This paper examined text coverage using CEFR-based wordlist and syntactic complexity using the Python library spaCy. Findings revealed that the proportion of A2 level tokens does not differ greatly between AI-generated passages (ChatGPT: 19.1%; Gemini: 17.4%; DeepSeek: 17.3%) and ESCT (15.6%); ESCT had more complex but shorter sentences comparing to AI-generated passages, and personal pronouns accounted for 1.3% of ESCT tokens, while they accounted for less than 1% of generative AIs tokens (ChatGPT, 0.39% ; Gemini, 0.71% ; DeepSeek 0.45%). Few wh-pronouns and the existential there were found in AI-generated passages. This study concluded that when the EFL teachers convert AI-generative text to the mock ESCT, they should (a) add a wider variety of A1 level lemmas, (b) rewrite more complex and shorter sentences, (c) increase personal pronouns and determiners, and (d) reduce adjective modifiers, conjuncts, and coordination.