Multi-Modal Sarcasm Detection via Knowledge-Aware Focused Graph Convolutional Networks

Multi-Modal Sarcasm Detection (MSD) aims to combine multiple modal information to identify implicit sarcastic sentiment. However, the significance of commonsense knowledge in implicit emotion recognition has been frequently overlooked. Additionally, the important visual emotions associated with textual modal sarcastic cues are typically dispersed throughout the entire image, making it difficult to accurately focus on crucial sarcastic features. Therefore, we propose a Knowledge-Aware Focused Graph Convolutional Networks (KFGC-Net) to tackle these issues. Specifically, we first construct a cross-modal knowledge-aware graph based on commonsense concepts for each instance. This graph explicitly establishes connections between significant visual sentiments and relevant textual tokens of knowledge. Next, we integrate the transformer encoder optimized with convolutional operations with the Convolutional Block Attention Module to compensate for the model’s lack of attention to important features. Finally, we design a Global Modality Synergistic Fusion (GMSF) block, aiming to model the global relationships in each modality for complementing global sarcasm detection result. Notably, we analyze the proposed framework by testing it on several benchmark datasets, and the results outperform the existing state of the art.