Exploring the Potential of Generative AI in Essay-Based Assessments: Evidence from the Regents Exam Data

This study empirically analyzed the potential of generative AI in constructed-response assessment by comparing the scoring performance of GPT-4o, Gemini 2.0, and Gemini 2.5 on written responses from the New York State Regents Examinations. Analyses using the Quadratic Weighted Kappa (QWK), Mean Absolute Error (MAE), and Pearson Correlation Coefficient (PCC) showed that all models achieved high accuracy, with QWK scores ranging from 0.889 to 0.935, MAE from 0.210 to 0.410, and PCC from 0.904 to 0.944. Gemini 2.5 performed best on evidence-based argument tasks, while GPT-4o showed the highest accuracy on text-analysis items. Confusion matrix analysis revealed that most errors were within ±1 point, though some limitations were observed, including boundary-level misclassifications and overestimation of zero scores. By employing a refined scoring rubric and grade-specific anchor papers as preparatory materials, and by implementing system prompts to mitigate large language model bias, this study distinguishes itself from prior research. These findings suggest that generative AI can serve as a reliable tool for evaluating constructed responses and propose a collaborative human-AI scoring framework.