A large-scale comparison of human and large language model raters in automated essay scoring

Large language models (LLMs) have brought about pivotal changes in writing assessment, particularly through comparisons with human raters. Nevertheless, most current studies are limited to analyzes that compare a small number of models with human raters and focus primarily on overall score differences or inter-rater agreement. This quantitative study compares 17 cutting-edge LLMs on the same dataset, examining the models’ scoring consistency, proximity to the mean scores assigned by human raters, and structural similarity and clustering analyzes of scoring behaviours relative to those of humans, based on analytic sub-dimensions, using a multidimensional approach. To that end, the research analyzed literary essays written by 46 first-year students at upper-intermediate proficiency in the English Language and Literature department of a state university in Türkiye, based on Ernest Hemingway's short story ‘The Cat in the Rain,’ using an expert-validated six-dimensional analytic rubric. The findings reveal substantial variability at the level of analytic sub-dimensions among 17 LLMs despite their proximity to experts in terms of overall scores. Moreover, these models clustered according to their average dimension-score profiles, while some displayed closer alignment with human rater profiles. Finally, while different correlation values were identified for each analytic dimension, high consistency was noted between the two expert raters with respect to overall scoring.