HintsOfTruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claims

Misinformation can be countered with factchecking, but the process is costly and slow.Identifying checkworthy claims is the first step, where automation can help scale fact-checkers' efforts.However, detection methods struggle with content that is (1) multimodal, (2) from diverse domains, and (3) synthetic.We introduce HINTSOFTRUTH, a public dataset for multimodal checkworthiness detection with 27K real-world and synthetic image/claim pairs.The mix of real and synthetic data makes this dataset unique and ideal for benchmarking detection methods.We compare fine-tuned and prompted Large Language Models (LLMs).We find that well-configured lightweight textbased encoders perform comparably to multimodal models but the former only focus on identifying non-claim-like content.Multimodal LLMs can be more accurate but come at a significant computational cost, making them impractical for large-scale applications.When faced with synthetic data, multimodal models perform more robustly.https://hintsoftruth.github.io/