SR-CIBN: Semantic relationship-based consistency and inconsistency balancing network for multimodal fake news detection

The fast dissemination of online information has facilitated the evolution of multimodal fake news, thereby rendering the trustworthiness of its content difficult to identify. Some existing studies, although capturing the consistency and inconsistency features between different modalities, neglect to dynamically balance these two types of features based on their contributions during the features fusion. Thus, we propose a S emantic R elationship-based C onsistency and I nconsistency B alancing N etwork for multimodal fake news detection (SR-CIBN). Specifically, the global features are thoroughly investigated by hierarchically penetrating and interacting between multimodal features that are aligned by contrastive learning at both the intra- and inter-modal views. Then, the global consistency and inconsistency features are obtained through the interaction between the selected key image patches features and the global features. Additionally, the fusion intensity of the global consistency and inconsistency features is adjusted based on the image–text matching degree, resulting in the final optimized features. Under the joint learning framework we proposed, confusion between semantically similar real and fake news is effectively avoided by training with triplet loss based on the image–text semantic relationship. Our model surpasses comparable approaches, as shown by comprehensive experiments on the Twitter and Weibo datasets.