Improving AI-assisted grading reliability through explicit rubric and prompt engineering
The use of artificial intelligence (AI) to support assessment and feedback in higher education has attracted growing interest, but concerns remain about the consistency and appropriateness of AI-generated grades in summative contexts. This study examined whether a combined calibration strategy involving a more explicit rubric and marking prompt could improve agreement between AI-assisted grading and educator judgement in a laboratory-based STEM assessment. We compared educator-assigned marks for 40 graduate Electrochemical Energy Storage Systems laboratory reports with marks generated by GPT-5 under pre- and post-calibration configurations, and examined agreement using complementary statistical and visual analyses. Following calibration, agreement with educator marks improved across measures of score correspondence and absolute agreement: the intraclass correlation coefficient increased from 0.818 to 0.928, the mean AI-educator difference decreased from +1.625 to +1.041 marks, and no post-calibration report differed from the educator mark by more than 10 marks. These findings suggest that the way assessment criteria, evidence requirements, and marking rules are specified can materially influence AI-human grading agreement. However, the revised rubric and prompt were introduced together, calibration drew on discrepancies observed in the same 40 reports used for post-calibration evaluation, and one educator's marks served as the reference. The results therefore support improved in-sample agreement under a human-supervised calibration strategy, rather than establishing grading validity or interchangeability for autonomous summative decisions.
Authors
- Deepshikha Deepshikha
- Navpreet Singh
Institutions
- Queen Mary University of London (GB)
- Cranfield University (GB)
Publication Details
- Journal
- Frontiers in Education
- Published
- 2026-11-18
- DOI
- https://doi.org/10.3389/feduc.2026.1968847
- Primary Topic
- Artificial Intelligence in Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00