Enhancing graduate students’ evaluative skills through AI-supported collaborative marking

Developing reliable evaluative judgment is a central and cognitively demanding task in graduate education, as assessing research writing requires integrating multiple quality criteria while regulating subjective bias. Effective evaluation relies on learners' ability to internalize standards, monitor judgment accuracy, and calibrate decisions against expert benchmarks. Although prior AI-assisted assessment research has predominantly focused on optimizing predictive accuracy and scoring agreement with human raters, comparatively less attention has been paid to how AI-supported marking may function as a psychologically mediated process shaping learners' evaluative judgment. Grounded in evaluative judgment theory and informed by self-regulated learning perspectives, this study conceptualizes AI-supported collaborative marking as a metacognitive scaffold that externalizes expert criteria, facilitates discrepancy detection, and supports reflective calibration. Using a mixed-method approach, 121 graduate research papers were assessed under four conditions: student-only marking, AI-only marking, retrieval-augmented AI marking, and AI-supported collaborative marking. Papers were evaluated across four dimensions including rigor, originality, significance, and academic conventions, with outcomes benchmarked against expert judgments. Results showed that AI-supported collaborative marking reduced deviations from expert ratings and improved inter-rater consistency compared with student-only and AI-only conditions, with MAE decreasing from 3.260 to 1.221. Behavioral sequence analyses revealed systematic differences in evaluative behaviors. Senior graduate students exhibited more developed metacognitive monitoring, reflective reasoning, and structured cycles of planning, monitoring, and reflection. In contrast, junior students relied more on exploratory and trial-and-error strategies, highlighting developmental differences in self-regulated evaluative competence. Overall, the findings indicate that AI-supported collaborative marking enhances assessment accuracy, and is associated with observable changes in students' evaluative interaction patterns and reflective behaviors, which may contribute to the development of academic assessment literacy in graduate education.