When alignment is not agreement: Construct-specific validation of large language models for CAF measurement in L2 writing

Abstract Although current research shows that large language models (LLMs) have become more popular in second-language (L2) writing assessment, we must note that strong correlation with human raters does not imply interchangeable measurement. This study assessed the ability of six public facing LLMs to replicate values for explicit complexity, accuracy and fluency (CAF) obtained from reviewed human consensus coding. There were 90 observations of Iranian EFL undergraduates' argumentative essay, which were produced by 45 students in class. Three measures of product-based CAF measures were used: words per T-unit for Fluency, S-nodes per T-unit for Complexity, and error-free T-units per T-unit for Accuracy. A single coding prompt and anonymized essays were each given to ChatGPT, Gemini, Grok, DeepSeek, Claude and Microsoft Copilot. Validation evidence included correlations, paired mean differences, bias, RMSE, normalized RMSE, Bland–Altman-type limits of agreement and within-writer change diagnostics. For all 18 human-LLM correlations, the results were positive and significant (p < .001); however, there was some variability across constructs and models. There was clear systematic underestimation with complexity, and the lowest participant-level change agreement, and accuracy was best reproduced. DeepSeek performed the best overall approximation using the combined criteria, and ChatGPT showed the best accuracy approximation. On tasks, human reference was more accurate but less fluent and complex in the essay two; models reproduced accuracy directions more consistently than the fluency and complexity directions. The results indicate that alignment is not interchangeable and suggests construct-specific validity, transparent prompting, and human verification for L2 writing assessment using LLMs.