Comparative Analysis of Reasoning Capabilities in Foundation Models

The rapid advancement of large language models (LLMs) has led to significant improvements in natural language processing tasks, yet their capability for complex reasoning across diverse domains remains understudied. This paper presents a comprehensive evaluation of the reasoning abilities of six state-of-the-art LLMs: Mixtral-8x7B, Phi-3, LLaMA 3.1-8B, Gemma-2-9b, Mistral-7B, and OLMo-7B. We employ a novel framework to assess these models across eight academic domains, including Physics, Mathematics, Chemistry, and Economics, with problems of varying difficulty levels. Leveraging the MareNostrum 5 supercomputer, our study provides a comprehensive evaluation, offering insights into the models’ performance, consistency, and step-by-step reasoning capabilities. Results reveal significant variations in performance, with Mixtral-8x7B and Phi-3 demonstrating superior overall capabilities, while domain-specific strengths emerged across different models. Notably, all models excelled in Economics but struggled with Optimization problems. Analysis of step accuracy highlighted intriguing differences in problem-solving approaches, with Gemma-2-9b showing exceptional transparency in its reasoning process. Our findings highlight the complex relationship between model architecture, training methodologies, and reasoning abilities, challenging the notion that larger models invariably perform better. This study contributes valuable insights for the development and application of AI in complex problem-solving tasks, emphasizing the need for balanced capabilities across domains and transparent reasoning processes in future LLM development.