Analyzing the Mathematical Proficiency of Large Language Models in Computer Science Graduate Admission Tests

This study evaluates the performance of six prominent Large Language Models (LLMs) on graduate entrance exam multiple-choice mathematics questions in computer science, computer engineering, and information technology programs, with a focus on their cross-lingual capabilities. The selected models, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1 405B, Mistral Large 2, and Qwen 2.5 72B, were tested on 146 questions presented in both Persian and English, spanning four key mathematical domains. Results reveal significant variations in accuracy, with Gemini 1.5 Pro achieving the highest overall performance in English (63.70%) and Claude 3.5 leading in Persian (52.0%). However, some models struggled with maintaining consistent accuracy across languages, showing a cross-lingual performance gap. The findings underscore the potential of LLMs in addressing complex mathematical tasks but also highlight their current limitations, particularly in multilingual contexts. Notable disparities in model performance point to the importance of architectural innovations and multilingual training.