Analysis of mathematical performance and reasoning process of generative AI: focusing on the college scholastic ability test

This study analyzes the mathematical performance and reasoning processes of the generative AI model o1-preview to explore its potential application in mathematics education. In particular, this research uniquely delves into the reasoning processes of generative AI, identifying specific stages and characteristics that emerge during problem-solving. Using math problems from the Korean College Scholastic Ability Test (2022-2024), the study examined the model's response accuracy and the characteristics of its reasoning steps. The findings reveal that o1-preview demonstrated high accuracy in 2-point and 3-point questions and successfully solved some challenging 4-point questions. During the reasoning process, the model predominantly derived correct answers through the ‘analysis’ and ‘planning/execution’ stages, with particularly high accuracy observed when the ‘exploration’ and ‘verification’ stages were included. This study suggests that o1-preview could serve as a valuable computational tool in mathematics education, highlighting its potential for practical application. However, the study also identified limitations in the model's performance, particularly when ‘exploration’ and ‘verification’ stages were insufficiently executed. To address these issues, the importance of teacher guidance and well-designed prompts for utilizing generative AI in problem-solving is emphasized. This research evaluates the practical applicability of generative AI in real educational settings and aims to explore effective usage strategies for diverse educational environments in the future.