AGCL: Attention Aggregation Contrastive Learning for Multimodal Emotion Analysis

Multimodal sentiment analysis aims to predict the emotions in videos, which requires integrating heterogeneous information from the visual, auditory, and textual modalities and achieving effective feature fusion. Fusion methods based on attention mechanisms have gained widespread attention due to their outstanding performance and strong robustness. However, recent research indicates that these methods heavily rely on good feature representations in multimodal data. To address this issue, we propose the AGCL model, which uses an attention aggregation mechanism to fuse the visual and auditory modalities, enhancing semantic richness and bridging the gap in information density with the textual modality, ensuring the quality of modality information. Subsequently, hierarchical contrastive learning is implemented via attention mechanisms, which effectively decompose heterogeneous information into similar and dissimilar features, further enhancing the feature representation between the visual-auditory and textual modalities. Finally, we conducted extensive experiments on the CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets to evaluate various state-of-the-art multimodal sentiment analysis methods. The experimental results demonstrate that AGCL outperforms all baseline methods across various metrics on these datasets.