EUROPA Eval: A Parallel, Inspectable Capability Benchmark for Post-Training in 36 European Languages

Technical report, code, benchmark items and run artifacts for EUROPA Eval, an open multilingual capability benchmark for post-trained language models. It applies the same 12 task templates (reading, arithmetic, grounded answering and calibrated abstention, structured JSON, exact instruction following, tool-error recovery, a defined text calculation, summarization, translation, harmful-request refusal and revision tracking) to each of the 36 languages prioritized by OpenEuroLLM (432 items), with strict deterministic scoring and a pinned LLM judge. The report evaluates nine models, including five post-training variants of one 9B OpenEuroLLM base. The PDF is in paper/europa-eval.pdf.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24339070
Primary Topic
Natural Language Processing Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

EUROPA Eval: A Parallel, Inspectable Capability Benchmark for Post-Training in 36 European Languages

Birger Moëll
4 links
Zenodo (CERN European Organization for Nuclear Research)
Natural Language Processing Techniques
article

EUROPA Eval: A Parallel, Inspectable Capability Benchmark for Post-Training in 36 European Languages

Birger Moëll
article en

Abstract

Technical report, code, benchmark items and run artifacts for EUROPA Eval, an open multilingual capability benchmark for post-trained language models. It applies the same 12 task templates (reading, arithmetic, grounded answering and calibrated abstention, structured JSON, exact instruction following, tool-error recovery, a defined text calculation, summarization, translation, harmful-request refusal and revision tracking) to each of the 36 languages prioritized by OpenEuroLLM (432 items), with strict deterministic scoring and a pinned LLM judge. The report evaluates nine models, including five post-training variants of one 9B OpenEuroLLM base. The PDF is in paper/europa-eval.pdf.

Zenodo (CERN European Organization for Nuclear Research)
AI Sweden (SE)
Natural Language Processing Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon