Measuring Complexity and Reproducibility: A Comprehensive Benchmark of LLMs for Multilingual Policy Agenda Topic Annotation

Although Large Language Models (LLMs) have rapidly expanded the text-as-data toolkit available to social scientists, their performance remains highly sensitive to task complexity, prompting, language and model parameters. In order to assess these limits, we conduct a cross-lingual benchmark of 79 contemporary LLMs (e.g., OpenAI’s GPTs and o-series, Claude models, xAI’s Grok, Meta’s Llama, Alibaba’s Qwen, Mistral models, among others) on a demanding zero-shot classification task: labelling one of the 21 major topics in the Comparative Agendas Project codebook to bills and Acts drawn from Danish, Dutch, English, French, Hungarian, Italian, Portuguese and Spanish corpora. We then conducted a meta-analysis that shows reasoning models lift F1-scores by around nine percentage points on average. In comparison, reproducible models under deterministic deployment conditions incur a cost of about eight percentage points in F1-score. In addition, preliminary and ongoing fine-tuning experiments suggest that fine-tuned transformers may surpass the best zero-shot LLM classification performance. These findings quantify the trade-off between performance and reproducibility, highlighting chain-of-though reasoning as the most effective option for complex policy agenda annotation. They also set a baseline for future venues, where systematic fine-tuning and alignment strategies could be used to test whether some models can close the performance gap while retaining full reproducibility.