Can students judge like experts? A large-scale study on the pedagogical quality of AI and human personalized formative feedback

While feedback is essential for guiding student learning, providing timely and personalized guidance in large-scale educational settings remains a significant challenge. Generative AI offers a scalable solution, yet little is known about students’ perceptions of AI-generated feedback. In this paper, we aim to investigate how the identity of the feedback provider (human vs. AI) affects students’ ability to assess feedback quality and whether their judgments are biased. We propose a comprehensive rubric for assessing the pedagogical quality of formative feedback. We use it to compare the objective quality of AI-generated and human-crafted feedback (N = 979). Next, using data collected from 472 STEM students, we examine the extent to which students’ perceptions of the same feedback align with those of the experts. Our contribution is threefold. First, by introducing a structured rubric, we address the need for more standardized and reliable methods to assess the pedagogical quality of AI-generated feedback. Second, our analysis indicates that the pedagogical quality of AI-generated feedback is, in practice, comparable to that of human-authored feedback. However, both types exhibit limitations, particularly in addressing metacognitive aspects. Third, students’ evaluations are largely influenced by their perceptions of the feedback provider’s credibility rather than the actual quality of the feedback itself. This pattern is consistent across all academic levels, genders, and fields of study. Our findings underscore the need for targeted strategies to enhance students’ ability to evaluate feedback objectively and to improve the pedagogical quality of AI-generated feedback, thereby strengthening the effectiveness of AI-powered educational feedback systems. • We introduce a robust rubric to assess the pedagogical quality of formative feedback. • Based on 979 responses, AI and human feedback demonstrate comparable pedagogical quality. • Metacognitive elements are often absent in both AI and human feedback. • 472 STEM students across various levels and disciplines evaluated AI and human feedback. • Student ratings are influenced more by the credibility of the feedback source than by its content.