Comparative judgment and writing quality: reliability and validity across expert, peer, crowdsourced, and LLM-generated judgments

Prior research has demonstrated the strong reliability and concurrent validity of the comparative judgement (CJ) method for assessing writing quality. However, the factors considered when making these judgements are less investigated, as well as their potential variation across raters with different backgrounds. In Study 1, we asked four participant groups to rate the writing quality of 90 Arabic essays using the CJ method. Regression and correlational analysis showed that expert, crowdsource, and GPT-4o’s CJ ratings exhibited acceptable levels of reliability and concurrent validity, while peers demonstrated only some reliability. In Study 2, we analyzed decision comments left by three groups (experts, crowdsourced workers, peers) after making their judgments. We found evidence for construct validity among experts and crowdsourced workers. These results provide evidence supporting the reliability and validity of the CJ method across three distinct rater groups (experts, crowdsource workers, GPT-4o) and extend its application to a novel assessment context.