Exploring the Impact of Discussion Responses Generated From <scp>ChatGPT</scp> on Student Performance and Experiences

ABSTRACT This study explores how analyzing ChatGPT‐generated responses in mathematics discussions can enhance students' critical thinking skills and identify which types of discussions improve final exam scores. In a remedial mathematics course for STEM majors, we assess students' ability to evaluate the correctness of mathematical arguments, provide counterexamples, demonstrate conceptual understanding, and engage in meaningful discourse as they analyze AI‐generated responses. Using a mixed‐methods approach, quantitative data was collected from 40 students' scores on five discussion assignments and their final exam scores. Regression analysis revealed a significant model explaining approximately 40% of the variance in final exam scores, with discussions focused on real‐world applications and conceptual understanding being stronger predictors than those centered on procedural knowledge. Survey questionnaires provided insights into students' experiences analyzing ChatGPT‐generated responses. The findings suggest that analyzing ChatGPT‐generated responses may enhance students' critical thinking skills and a deeper understanding in mathematics teaching and learning, contributing to the ongoing exploration of AI integration in mathematics education as well as STEM education.