EuroBERT: Scaling Multilingual Encoders for European Languages

General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

Authors

Institutions

Publication Details

Journal
ArXiv.org
Published
2025-03-07
Primary Topic
Natural Language Processing Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

EuroBERT: Scaling Multilingual Encoders for European Languages

Duarte M. Alves, Ricardo Rei, Gabriel Hautreux, Caio Corro et al.
ArXiv.org
Natural Language Processing Techniques
article

EuroBERT: Scaling Multilingual Encoders for European Languages

Duarte M. Alves, Ricardo Rei, Gabriel Hautreux, Caio Corro, Nuno Guerreiro, André F. T. Martins, Fanny Jourdan, Ayoub Hammal, Patrick Fernandes, Nicolas Boizard, Céline Hudelot, Emmanuel Malherbe, Pierre Colombo, Maxime Peyrard, Manuel Faysse, Etienne Malaboeuf, Hippolyte Gisserot-Boukhlef, João Alves, Kevin El-Haddad
article en

Abstract

General-purpose multilingual vector representations, used in retrieval, regression and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively supporting sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

ArXiv.org
Supélec (FR), Total (France) (FR), Universidade Politécnica de Lisboa (PT), Université Paris-Saclay (FR), Institut National des Sciences Appliquées de Rennes (FR), Centre Informatique National de l'Enseignement Supérieur (FR), IRT M2P (FR), Laboratoire d'Informatique de Grenoble (FR), CentraleSupélec (FR), Artelys (France) (FR), GIPSA-Lab (FR), Mathématiques et Informatique pour la Complexité et les Systèmes
Openalex Percentile: Top 82%
Natural Language Processing Techniques
Open Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.