Benchmarking 24 Large Language Models for Automated Multiple-Choice Question Generation in Latvian

Abstract Large Language Models (LLMs) are increasingly being used for a wide range of text generation tasks. This paper investigates the generation of Multiple-Choice Questions in Latvian to assess both the ability of LLMs to generate high-quality questions and answers and, more broadly, their capability to process Latvian, a lower-resourced language that has received relatively little attention in LLM research. This study benchmarks 24 different LLMs, specifically those developed by Anthropic, DeepSeek, OpenAI, Google, Meta, Mistral, and Microsoft. The findings highlight the varying capabilities of these models in handling Latvian, producing grammatically correct, coherent, and meaningful text. The best-performing closed-weights model is claude-3.5-sonnet (by Anthropic), the best-performing open-weights model is deepseek-v3 (by DeepSeek), and the best-performing small open-weights model is open-mistral-nemo (by Mistral).