Study of Small and Medium-Scale Open-Source Language Models in Human-like logic and Comprehension Tasks

Logic and mathematics offer a uniquely rigorous framework for evaluating the reasoning capabilities of large language models (LLMs), as they rely on objective, verifiable answers. This study investigates the reasoning and comprehension abilities of several open-source language models executed locally with limited computational resources of a personal computer using the Ollama framework. Assessment of small models through these types of problems provides insights into the limits and potential of these models in achieving abstract reasoning, highlighting understanding at a small scale or requiring larger models. Two comprehensive experimental tasks were designed to evaluate performance: the first focuses on factual comprehension based on a healthcare context, and the second on solving logical problems. The first experiment consisted of factual yes/no questions derived from a publicly available article on the human cardiovascular system, while the logical problems involved short, multi-step reasoning tasks, including symbolic deduction, analogical reasoning, and conditional statements. To facilitate comparative analysis, the six evaluated models are grouped by their total number of trainable parameters: small-scale models (0.5 Billion to 3B parameters) versus medium-scale models (12.2 B to 16 B parameters). The small-scale models included LLaMA3 (1B) and (3B), Qwen2.5 (0.5B), and Phi (2.7B). The medium-scale models comprised Mistral-Nemo (12.2B), and DeepSeek-Coder-v2 (16B). Quantitative metrics including accuracy, precision, and recall were used to evaluate performance in Experiment 1, while a qualitative comprehension score was applied in Experiment 2. Many of the models tested locally have shown promising results and deserve to be further explored, but there are also models with very poor results in logical reasoning such as Phi. Our results show that Mistral-Nemo and DeepSeek-Coder-v2 perform best in factual comprehension, achieving accuracies of 96.67% and 86.67% respectively. Moreover, DeepSeek-Coder-v2 showed stronger results in logical reasoning compared to the other models. However, none of the models achieved maximum performance across all tasks, suggesting that there is still substantial room for improvement in reasoning capabilities. Furthermore, applying the chain-of-thought prompting technique, despite its effectiveness reported in prior studies, did not lead to consistent improvements across the evaluated small and medium-sized models, suggesting that its impact may be strongly dependent on model scale and capacity.