GPT-4 as an Automatic Grading System for Open-Ended Questions in Computer Science Education

This paper examines the potential of GPT-4 as an automated grading system for open-ended questions in computer science education, employing an extended version of Bloom’s Taxonomy as its theoretical framework. Traditionally, the evaluation of open-ended responses has relied on labor-intensive manual grading, which is not only time-consuming but also subject to variability in consistency and subjective bias. Recent advances in natural language processing, however, offer promising alternatives by enabling the automation of complex evaluative tasks. This study aims to investigate whether GPT-4 can serve as a reliable and scalable tool for formative assessments in educational environments, thereby reducing the workload on educators and providing timely, consistent feedback to students. A quantitative comparison between GPT-4’s evaluations and those of human experts reveals moderate alignment, with stronger consistency at lower cognitive levels (e.g., recall and application) and more pronounced discrepancies at higher levels (e.g., evaluation and creation). Statistical metrics, including Pearson’s correlation, Intraclass Correlation Coefficient (ICC), and Cosine Similarity, were employed to measure agreement. The findings demonstrate that GPT-4 can effectively maintain relative rankings of student performance, providing a scalable option for formative assessments. However, limitations remain in nuanced and context-specific evaluations, requiring further refinement for deployment in the evaluation process.