Reliable or Not: A Correlational Study of Automated Scoring and Analytic CAF Indices in A Computer-Based Monological English Speaking Test

The rise of computer-based testing has transformed language assessment, with computer-based English speaking tests (CBESTs) offering scalable and efficient evaluations. However, the extent to which automated scoring aligns with analytic measures, such as complexity, accuracy, and fluency (CAF) indices, remains unclear. This study examines the relationship between automated CBEST scores and CAF indices in 418 Chinese junior high school students’ spoken responses. Findings revealed strong correlations between automated accuracy and fluency scores and their respective analytic indices, demonstrating the system’s ability to assess grammatical precision and temporal fluency. However, automated complexity scores correlated only moderately with mean length of clauses, with no significant links to lexical diversity. Cross-dimensional correlations suggested potential overlap in scoring constructs. These results highlight CBESTs’ strengths and limitations in assessing oral proficiency, emphasizing the need to refine scoring algorithms to improve validity and comprehensiveness in automated language assessments.