Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models

Abstract This study investigates the potential of Generative AI models and sentence embedding models for the automated assessment of open-ended student responses in a higher education computer science course. Among 110 university students enrolled in a software engineering course, 1,885 responses to 24 open-ended questions assessing knowledge of software engineering concepts were collected. Using precision, recall, F1-score, false positive and false negative rates, and inter-rater agreement metrics such as Fleiss’ Kappa and Krippendorff’s Alpha, we systematically analyzed the performance of eleven state-of-the-art models, including GPTo1, Claude3, PaLM2, and SBERT, against two human expert graders. The findings reveal that GPTo1 achieved the highest agreement with human evaluations, showing almost perfect agreement, low false positive and false negative rates, and strong performance across all grade categories. Models such as Claude3 and PaLM2 demonstrated substantial agreement, excelling in higher-grade assessments but falling short in identifying failing grades. Sentence embedding models, while moderately effective, struggled with capturing the context and semantic nuances of diverse student expressions. The study also highlights the limitations of reference-based grading approaches, as shown by the Natural Language Inference analysis, which found that many student responses contradicted reference answers despite being semantically correct. This underscores the importance of context-sensitive models like GPTo1, which accurately evaluate diverse responses and ensure fairer grading. While GPTo1 stands out as a candidate for independent deployment, the financial cost of such high-performing proprietary models raises concerns about scalability.