Bilingual analysis of logical coherence and accuracy in large language model performance on high school math word problems: Russian <i>vs</i> . English

Nowadays, large language models (LLMs) have demonstrated impressive performance in solving math word problems (MWPs). However, there are currently challenges on their reasoning abilities, particularly on problems demanding advanced reasoning, with notable inconsistencies observed across different languages. This article conducts a thorough evaluation of the reasoning capabilities of LLMs like Gemini-1.5-Pro, Llama 3.1-406B-Instruct, Llama 4-Maverick-Instruct, Mistral-8x22B-Instruct-v0.1, ChatGPT-3.5 Turbo, ChatGPT-4, ChatGPT-4o on math word problems (MWPs) in Russian and English languages. Understanding the bilingual performance of LLMs in solving MWPs is crucial for assessing their potential and limitations in a global context. Despite their ability to generate correct answers, the coherence in logical inference and their accuracy is uncertain, particularly with complex math word problems. In response to this challenge, we introduce the QazUNTv2 dataset, featuring high school math word problems (MWPs) in Russian and English, along with categorized labels that represent sections in mathematics, such as algebra, probability theory, and logic, and meticulously crafted manual solutions. Further, solutions were obtained through LLMs for comparison purposes. To evaluate them against the dataset’s reference versions, we applied ROUGE-1 and ROUGE-2 (Recall-Oriented Understudy for Gisting Evaluation) and BERTScore (Bidirectional Encoder Representations from Transformers Score) metrics. We conducted a comparative analysis of LLM behavior when solving tasks from different mathematical domains mentioned before. The study examines the differences in LLM performance when solving mathematical problems in English and Russian, highlighting the impact of linguistic structure on reasoning accuracy. We identified seven primary types of reasoning errors and quantitatively assessed error rates across models. The results show that ChatGPT-3.5 Turbo exhibited the highest error rate in both languages and differs significantly from all other models ( p &lt; 0.001), while Gemini 1.5 Pro and GPT-4 Turbo demonstrated greater accuracy, emphasizing the need for improved logical inference in LLMs. Additionally, our findings reveal notable cross-linguistic differences in model responses, with Russian answers being more concise than English ones and incorrect responses tending to be more verbose than correct ones. These patterns suggest that linguistic structure influences reasoning efficiency, while verbosity may serve as an indicator of model uncertainty.