Evaluating visual mathematics in multimodal LLMs: a multilingual benchmark based on the Kangaroo tests

Abstract Multimodal Large Language Models (MLLMs) promise advanced vision-language capabilities, yet their effectiveness in visually presented mathematics remains underexplored. This paper analyses the development and evaluation of MLLMs for mathematical problem-solving, focusing on diagrams, multilingual text, and symbolic notation. The computational demands of evaluating these large-scale models across multilingual datasets necessitate high-performance computing infrastructure, as systematic benchmarking of state-of-the-art MLLMs requires distributed processing of thousands of inference requests and parallel evaluation across multiple model architectures. We then assess several models-including GPT-4o, Pixtral, Qwen-VL, Llama 3.2 Vision variants, and Gemini 2.0 Flash-in a multilingual Kangaroo-style benchmark spanning English, French, Spanish, and Catalan. Our experiments reveal four key findings. First, overall accuracy remains moderate across geometry, visual algebra, logic, patterns, and combinatorics: no single model excels in every topic. Second, whilst most models see improved accuracy with questions that do not have images, the gain is often limited; performance for some remains nearly unchanged without visual input, indicating underutilisation of diagrammatic information. Third, substantial variation exists across languages and difficulty levels: models frequently handle easier items but struggle with advanced geometry and combinatorial reasoning. Notably, Gemini 2.0 Flash achieves the highest accuracy on image-based tasks, followed by Qwen-VL 2.5 72B and GPT-4o, though none approach human-level performance. Fourth, a complementary analysis aimed at distinguishing whether models reason or simply recite reveals that Gemini and GPT-4o stand out for their structured reasoning and consistent accuracy. In contrast, Pixtral and Llama exhibit less consistent reasoning, often defaulting to heuristics or randomness when unable to align their outputs with the given answer options. Furthermore, detailed error analysis identifies two primary failure modes: encoding-stage errors, where models misidentify visual elements such as colours or shapes, and visio-semantic processing errors, where models struggle with three-dimensional spatial reasoning and geometric relationships, revealing systematic limitations even in state-of-the-art architectures.