Effects of structured generative AI feedback on data science students' critical thinking: a propensity score matching approach

Background AI feedback holds promise for supporting student learning, but unstructured use can lead students to passively accept AI-generated answers, weakening their independent reasoning. This study examined whether structured AI feedback affects undergraduate students' critical thinking, learning performance, and engagement in a data science analytics course. Methods An exploratory two-class quasi-experimental study was conducted in a three-month undergraduate data science course ( N = 167). One class received four-phase structured AI feedback, while the comparison class received conventional instructor feedback. Propensity score matching was used to improve comparability on observed baseline characteristics. Critical thinking was assessed using an adapted 40-item assessment covering five dimensions. Results Students receiving structured AI feedback demonstrated higher post-test critical thinking scores than matched controls, with a baseline-adjusted difference of 5.64 points (95% CI [3.62, 7.65], p < 0.001). A significant between-group difference in pre-to-post changes was also observed. Dimension-level analyses revealed significant differences in Evaluation of Arguments, Interpretation, Recognition of Assumptions, and Inference, while Deduction showed no significant difference. Higher learning performance, cognitive engagement, and behavioral engagement were also observed in the structured AI feedback group, whereas emotional engagement did not differ significantly between groups. Conclusion The findings provide exploratory evidence of differences in critical thinking and learning outcomes between the structured AI feedback and conventional instructor feedback classes. They highlight the potential relevance of feedback designs that encourage students to actively evaluate, question, and revise AI-supported analytical work, while underscoring the need for further studies that can isolate intervention-specific effects.