EUROPA Eval: A Parallel, Inspectable Capability Benchmark for Post-Training in 36 European Languages
Technical report, code, benchmark items and run artifacts for EUROPA Eval, an open multilingual capability benchmark for post-trained language models. It applies the same 12 task templates (reading, arithmetic, grounded answering and calibrated abstention, structured JSON, exact instruction following, tool-error recovery, a defined text calculation, summarization, translation, harmful-request refusal and revision tracking) to each of the 36 languages prioritized by OpenEuroLLM (432 items), with strict deterministic scoring and a pinned LLM judge. The report evaluates nine models, including five post-training variants of one 9B OpenEuroLLM base. The PDF is in paper/europa-eval.pdf.
Authors
- Birger Moëll (ORCID: https://orcid.org/0000-0002-6529-1211)
Institutions
- AI Sweden (SE)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24339070
- Primary Topic
- Natural Language Processing Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00