A speaking comparison of text analysis tools:Levels of agreement and disagreement

This paper explores agreement levels among text analysis tools and factors influencing text difficulty using 75 Grade 2 EIKEN texts. EIKEN is a Japanese proficiency test for high school graduates. Text analysis included word count, average sentence length, CEFR, CEFR-J levels (determined by Text Inspector and CVLA, respectively), Flesch Reading Ease Score, Flesch-Kincaid Grade Level, Lexile score, and total coverage (from AntWordProfiler) using the New General Service List (NGSL). Average sentence lengths had stronger correlations with other indices than word counts. Flesch-Kincaid Grade Level and Lexile reading levels correlated moderately due to reliance on sentence length. According to both tools, the texts are generally deemed appropriate for grades 8-9 in the US education system. Considering that text levels for second language classrooms are typically several levels lower than those in the US education system and recognizing that EIKEN Grade 2 is intended for high school graduates in Japan, it can be inferred that the texts align with expectations for high school graduates in Japan, equivalent to grades 8-9 in the US education system. CEFR and CEFR-J levels, although their text level assignments were similar, had only moderate correlations, reflecting metric differences between Text Inspector and CVLA. AntWordProfiler's total coverage showed weak correlations, focusing solely on word frequency. The results from this study show clear discrepancies in text difficulty depending on the type of measure used and call for varied approaches to text analysis. Keywords: text analysis, EIKEN, text difficulty, text measures, Text Inspector, CVLA, AntWordProfiler, NGSL