MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages.Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for lowresource languages.To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages-over 100 more than the most multilingual existing VL benchmarks encompass.We then benchmark a range of open-weight LVLMs together with GPT-4o(mini) on MVL-SIB.Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo.Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance.We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multiimage tasks.By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs. 1 * Equal contribution. 1 Code: https://github.com/floschne/mvl-sibImages-To-Sentences (I2S)Which sentence best matches the topic of the images?The images and the sentences each belong to one of the following topics: "entertainment", "geography", "health", "politics", "science and technology", "sports", or "travel".Choose one sentence from A, B, C, or D.