Performance Analysis of Chinese Large Language Models in Solving Math Word Problems

Recently, researchers in the field of math word problem (MWP) solving have reported performance metrics for various large language models (LLMs) on benchmark datasets, with some models consistently achieving new state-of-the-art (SOTA) results. However, the benchmark datasets are often selectively curated and subjectively refined, leading to discrepancies between the evaluation metrics and actual experiences in real-world applications. To address this issue, we constructed a comprehensive dataset named AS-Math5k to assess more authentic and reliable performance. AS-Math5k is derived from practical teaching scenarios and encompasses multiple-choice questions and word problems across K-12 grade levels. Subsequently, we developed a unified evaluation framework that includes a prompt engineering module, a problem-solving module, and a result analysis module. This framework evaluates the performance of representative Chinese LLMs on AS-Math5k and provides a detailed discussion of performance variations among LLMs when solving problems of different grade levels and types. Experimental results indicate the following: (1) There is a notable discrepancy in the performance of large language models on real-world MWPs compared to benchmark datasets; (2) LLMs generally perform better on objective problems with clear, standardized answers; and (3) LLMs demonstrate strong comprehension and problem-solving abilities in both explicit relations and logical scenarios.