Cross-modal Consistency Reasoning with Large Language Models for Short Video-based Fake news Detection

The existing short video-based fake news detection methods are devoted to deeply capture different modal features and establishing their effective fusion to improve detection performance, and have made considerable progress. However, due to the excessive redundant information in videos, these methods struggle to learn effective key semantics. In addition, the current fusion strategies are relatively shallow, making it difficult to highlight valuable cross-modal consistency clues. To address that, we propose CRLM, a Cross-modal Consistency Reasoning with Large Language Model(LLM) framework for short video-based fake news detection. Specifically, we design a LLM-guided Consistency Learning module to capture the factual and commonsense consistency within various textual elements of fake news, thereby mitigating the impact of redundant information in videos. Meanwhile, we develop a cross-modal multi-view reasoning module to evaluate the consistency across modality pairs, thus enhancing the effectiveness of multi-modal semantic alignment and fusion. Finally, comprehensive experiments on two widely used datasets, FakeSV and FakeTT, prove the advantage of CRLM in short video-based fake news detection. Moreover, we employ a decision backtracking strategy to quantify the degree of influence of input information from different modalities on the final decision, so as to improve the interpretability of the framework.