Evaluating LLMs on Italian through multi-prompt: the Evalita-LLM benchmark

Abstract We present Evalita-LLM, a novel benchmark designed to evaluate Large Language Models (LLMs) on Italian tasks. The key innovation of Evalita-LLM is the use of multiple prompts for each task, which mitigates the model’s sensitivity to specific prompt formulations and ensures a robust and objective performance assessment. We propose an iterative methodology for selecting and validating multiple prompts, and metrics that capture both high performance and stability of a prompt given a set of LLMs. The resulting Evalita-LLM multi-prompt benchmark is based on ten native Italian tasks, eliminating translation issues and potential cultural biases. It includes a mix of multiple-choice and generative tasks, enabling more natural interactions with LLMs. We report experimental results from the benchmark’s development phase, along with performance statistics for several state-of-the-art LLMs. Evalita-LLM is publicly available with a leaderboard hosted on Hugging Face to support transparent evaluation and comparison of LLMs.