Exposing the Achilles’ Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning

Large Language Models (LLMs) have significantly impacted the field of Math Word Problems (MWPs), transforming how these problems are approached and solved, particularly in educational contexts.However, existing evaluations often focus on final accuracy, neglecting the critical aspect of reasoning capabilities.This work addresses that gap by evaluating LLMs' abilities to detect and correct reasoning mistakes.We present a novel dataset, MWP-MISTAKE, containing MWPs with both correct and incorrect reasoning steps generated through rule-based methods and smaller language models.Our comprehensive benchmarking of state-of-the-art models such as GPT-4o and GPT4 uncovers important insights into their strengths and limitations.While GPT-4o excels in mistake detection and rectification, gaps remain, particularly in handling complex datasets and novel problems.Additionally, we identify concerns with data contamination and memorization, which affect LLM reliability in real-world applications.While OpenAI' O1 model demonstrates 90% accuracy in reasoning and final answers on complex tasks, it remains weak in mistake detection.Our findings highlight the need for improved reasoning evaluations and suggest ways to enhance LLM generalization and robustness in math problemsolving. Question:What is the product of the coordinates of the midpoint of a line segment with endpoints at (1,1) and (-7,5)? Reasoning Step:The midpoint of a line segment is located at the point (h,k), where h = (x1 + x2)/2 and k = (y1 + y2)/2.In this case, the endpoints of the line segment are (1,1) and(-7,5).So, we can find the midpoint by calculating h = (1 + (-7))/2 = 6 and k = (1 + 5)/2 = 3.Therefore, the product of the coordinates of the midpoint is hk = 6 * 3 = 18. Reasoning chain Correct (Yes/No): No Corrected reasoning chain or NA:The midpoint of a line segment is located at the point (h,k), where h = (x1 + x2 )/ 2 and k = (y1 + y2)/2.In this case, the endpoints of the line segment are (1,1) and (-7,5).So, we can find the midpoint by calculating: h = (1 + (-7))/2 = -6/2 = -3 k = (1 + 5)/2 = 6/2 = 3 Therefore, the product of the coordinates of the midpoint is: hk = -3 * 3 = -9Final answer (just the number): -9Reasoning chain Correct (Yes/No):