Human and AI Judgment in EFL Speaking Assessment: Psychometric Dependability, Feedback Quantity and Coverage, and Exploratory Teacher Perspectives

This study examined human-AI judgment in EFL speaking assessment as a measurement problem relevant to research on individual differences: how consistently do expert raters and repeated AI sessions preserve between-person differences in a complex performance? Thirty oral-performance samples from first-year Chinese university English majors were evaluated by four experienced EFL teachers and by ChatGPT-4o across four repeated sessions using the same holistic and accuracy-fluency-complexity framework. Score distributions were summarized descriptively; human-rater intraclass correlations and generalizability-theory G- and D-studies evaluated dependability. Feedback quantity and thematic coverage were summarized descriptively, and post-results interviews with the same four teacher-raters were analyzed as exploratory participant reflections. Teachers produced more dependable single ratings. Holistic phi coefficients were 0.54 for one ChatGPT-4o session and 0.65 for one teacher, increasing to 0.83 and 0.88 with four assessments. Analytic single-assessment coefficients ranged from 0.54 to 0.56 for ChatGPT-4o and 0.61 to 0.65 for teachers. Across holistic and analytic scoring, a phi coefficient of at least 0.80 was reached with four repeated ChatGPT-4o sessions but three independent teachers. ChatGPT-4o’s analytic dimensions showed limited differentiation from its holistic scoring: accuracy matched the holistic score in all 120 AI ratings, while fluency and complexity differed in only three and four ratings, respectively. ChatGPT-4o also produced more rubric-relevant comments descriptively, while the four teacher-raters emphasized human oversight for contextual and interpersonal interpretation. The study contributes to intelligence research at the level of psychometric measurement and human-AI judgment: assessor systems can differentially preserve observed individual differences in complex performance. EFL speaking was not treated as an intelligence test, and unmeasured cognitive mechanisms were not inferred from the score patterns.