Who Grades More Consistently? Exploring AI vs. Human Teachers in the Assessment of Reflective Writing

Abstract The assessment of reflective writing remains challenging due to the subjective nature of the task. This exploratory study compares the assessment of students’ reflective writing by six human higher education teachers and an AI tool, ChatGPT-4o. Using rubrics, 13 reflections from students participating in a three-week internship programme were assessed across four key categories: knowledge and application, critical thinking, personal development, and clarity. Fleiss’ Kappa, correlation analysis, and independent t-tests were conducted to evaluate inter-rater reliability, grading consistency, and alignment between human and AI grading, while qualitative analysis was used to understand any grading discrepancies. Findings reveal that the introduction of AI as an additional grader did not significantly alter overall grading consistency, though it exhibited more structured grading patterns with lower variability compared to human assessors. Qualitative analyses suggest that AI prioritises coherence and linguistic clarity, potentially overlooking deeper critical reflections. This study highlights the complexities of evaluating reflective writing and the potential of AI-assisted grading tools to complement human-made assessments. Findings offer insights into the evolving role of AI in assessment and the need for a balanced, hybrid grading approach that integrates both human judgment and AI’s efficiency. Keywords Artificial intelligence, reflective writing, assessment, AI-assisted grading, assessment of reflection, reflection literacy