An In-Depth Survey on Multimodal Automatic Fact-Checking Datasets

Abstract The rapid spread of misinformation poses a significant challenge in the digital age, with false claims appearing across multiple modalities, particularly combinations of textual and visual content such as images and videos. While automatic fact-checking plays a crucial role in countering misinformation, traditional approaches predominantly rely on textual data, often neglecting the multimodal nature of modern misinformation. In this survey, we provide a comprehensive evaluation of multimodal datasets designed for automatic fact-checking that combine textual and visual information, systematically analyzing their sources, annotation methodologies, and key statistical properties, such as class distribution, topic diversity, and label availability. Additionally, we assess the usability of these datasets in real-world scenarios, discussing their limitations, biases, and potential risks, such as information leakage. Motivated by the practical difficulties we encountered when attempting to integrate existing datasets into our own multimodal fact-checking pipeline, our work also offers concrete guidance to help researchers choose the most suitable resources. By identifying gaps and challenges in existing datasets, our survey aims to support the development of more reliable and scalable multimodal fact-checking systems.