INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (\ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.
Authors
- Arshia Soltani Moakhar
- Christopher Klamm (ORCID: https://orcid.org/0009-0003-4342-0849)
- Swati Rajwal (ORCID: https://orcid.org/0000-0002-3826-5069)
- Börje F. Karlsson (ORCID: https://orcid.org/0000-0001-8925-360X)
- Azmine Toushik Wasi (ORCID: https://orcid.org/0000-0001-9509-5804)
- Imanol Schlag
- Zeming Chen (ORCID: https://orcid.org/0000-0002-2389-6968)
- Selvan Sunitha Ravi
- Ayush K Tarun (ORCID: https://orcid.org/0000-0003-1116-7597)
- Mike Zhang (ORCID: https://orcid.org/0000-0003-1218-5201)
- Negar Foroutan
- Gabriel Adriano de Melo (ORCID: https://orcid.org/0000-0001-5878-7967)
- Joseph Marvin Imperial (ORCID: https://orcid.org/0000-0003-1073-6129)
- Serhan Yılmaz (ORCID: https://orcid.org/0000-0003-4669-9593)
- Debjit Paul
- Esther Ploeger (ORCID: https://orcid.org/0009-0007-9525-3359)
- Jekaterina Novikova (ORCID: https://orcid.org/0000-0003-4754-6126)
- Shivalika Singh
- Roshan Santhosh
- Michael Chang (ORCID: https://orcid.org/0000-0002-5153-3697)
- Joel Niklaus (ORCID: https://orcid.org/0000-0002-2779-1653)
- Fajri Koto
- Dominik Krzemiński (ORCID: https://orcid.org/0000-0003-4568-0583)
- Jenny Chim
- Drishti Sharma (ORCID: https://orcid.org/0000-0001-5908-6898)
- Marzieh Fadaee (ORCID: https://orcid.org/0000-0002-4447-1213)
- Danylo Boiko
- Syrielle Montariol (ORCID: https://orcid.org/0000-0003-1355-8778)
- Sulav Duwal (ORCID: https://orcid.org/0000-0002-8772-6706)
- Sara Hooker (ORCID: https://orcid.org/0000-0002-0190-6459)
- Marjana Prifti Skënduli (ORCID: https://orcid.org/0000-0002-2707-1621)
- Rami Tamir
- Sree Harsha Nelaturu
- Antoine Bosselut (ORCID: https://orcid.org/0000-0001-8968-9649)
- É. H. Khalilov (ORCID: https://orcid.org/0000-0001-7603-5072)
- Micol Altomare
- Viraat Aryabumi
- Alfonso Amayuelas
- Nan Yu (ORCID: https://orcid.org/0000-0002-0177-2067)
- Shayekh Bin Islam
- Golda Cohen (ORCID: https://orcid.org/0000-0002-3220-1160)
- Anna Sotnikova
- Angelika Romanou
- Rishabh Maheshwary
- Jebish Purbey
- A Snegha
- Aditya Kumar Dalmia
- Abraham Diress
- Daniil Dzenhaliou
- Daniel Fernando Erazo Florez
- Fabian Farestam
- Perttu Isotalo
- Maral Jabbarishiviari
- Johan Samir Obando Ceron
- Sara Rydell
- Bardia Soltani Moakhar
- Thenuka Ovin Weerasinghe
- Mohamed A. Haggag
Institutions
- Aalborg University (DK)
Publication Details
- Journal
- VBN Forskningsportal (Aalborg Universitet)
- Published
- 2024-11-29
- DOI
- https://doi.org/10.48550/arxiv.2411.19799
- Citations
- 3
- Primary Topic
- Topic Modeling
- Type
- article
Funders
- National Science Foundation
- Allen Institute for Artificial Intelligence
- École Polytechnique Fédérale de Lausanne
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
- Innosuisse - Schweizerische Agentur für Innovationsförderung
- Centro Svizzero di Calcolo Scientifico
- National Supercomputing Centre Singapore