Matematisk problemlösning på italienska med stora språkmodeller

The thesis presents an evaluation of the mathematical reasoning capabilities of state-of-the-art Large Language Models (LLMs) using the Italian Language. Most of the LLMs’ investigations are carried out in English. Using a different language helps to assess how well these models are able to generalize their reasoning capabilities. Starting from a slightly modified translation of the dataset GSM8K used as a baseline in this work, five more new datasets are introduced in order to test the models in different situations. Each variant is defined to test the models’ behavior in case irrelevant information is present in the problem statement or in case the reasoning needed to answer the question becomes more complex. In all the experiments models were prompted using few-shot Chain-of-Thought technique to ensure a fair comparison. The performance of the models is accurately measured for every dataset and a detailed statistical analysis is conducted to identify significant trends. Some more advanced models such as DeepSeek-R1 were able to relatively retain a certain level of accuracy across the different datasets while other models such as Gemma2-9B performed significantly worse when faced with increasingly complex reasoning tasks. Furthermore, the statistical analysis revealed a significant negative correlation between answer length and correctness in case irrelevant information was added to the problem. The results collected also suggest that the addition of irrelevant information to the problem statement tends to impair model accuracy more than increased reasoning complexity. This work provides an overview of the strengths and the weaknesses of LLMs by evaluating these in different situations highlighting how performance can evolve based on the type of the problem presented.