FACT-CLIP: Fake News Detection via CLIP-Based Cross-Modal Attention and Transformer Fusion

Multimodal fake news detection has gained significant attention in recent years by many researchers. Numerous existing methods incorporate different attention mechanisms to enhance the fusion of multimodal features. However existing studies underutilized visual information and do not fully capture the semantic interactions within the multimodal data. Moreover, the potential for more advanced pre-trained multimodal feature learning models for fake news detection is less explored. In this paper, we propose FACT-CLIP a CLIP-based multimodal fake news detection approach that utilizes two-level visual feature representations to improve image utilization. We form multi-modal features by concatenating CLIP-generated features and obtaining cross-modal similarity between the two modalities. We introduce an attention mechanism to dynamically assign weights and integrate features. Furthermore, we introduce a custom transformer-based fusion method to capture semantic relationships between modalities and multimodal fusion. We perform extensive experiments on a popular multimodal dataset which shows that our model significantly outperforms state-of-the-art methods with an improvement of 2.8%.