Empowering students to critically validate AI-generated mathematical solutions through the rational questioning approach

Abstract Generative AI makes it easy for students to obtain problem-solving solutions in mathematics, but it also raises the risk of uncritical acceptance. This critical case study develops and examines a rational questioning approach, grounded in Habermas’s (1998) theory of rational behavior, to scaffold students’ validation of AI-generated solutions in a college calculus course. Twelve STEM undergraduates engaged in AI validation activities across one semester. Data included ChatGPT chat histories and written lab justifications from an early lab (pre-rational questioning) and a later lab (post-rational questioning). A microanalysis focused on three aspects: (a) the rationality criteria students applied, (b) the follow-up questions they posed to the AI, and (c) the presence of explicit warrants in their justifications. Findings showed a shift from step-checking to criterion-guided validation guided by rationality components. Students not only attended to content and method correctness but also considered method efficiency and articulated warrants. Students also posed criterion-aligned follow-up questions that fostered rational discourse with the AI, and their written justifications more often included explicitly mathematically grounded warrants. The findings suggest that the rational questioning approach may support students’ validation across three analytic aspects while also highlighting the need for further refinement, particularly with respect to communicative rationality. Implications for instruction and directions for future research are discussed, and potential risks of using AI in mathematics teaching are highlighted.