CMSA: Addressing semantic discrepancy and context dependency in multimodal sentiment analysis

Multimodal sentiment analysis (MSA) has emerged as a powerful approach to better understand human emotions by integrating information from multiple modalities such as text, audio, and visual data. However, challenges arise from semantic discrepancies and modality conflicts during multimodal fusion, as well as complex sentiment scenes. This study proposes a novel framework for addressing these issues, referred to as Context-aware Multimodal Sentiment Analysis (CMSA), which incorporates dynamic correction mechanisms and context-aware adaptation to enhance sentiment analysis accuracy. Specifically, we introduce a Text-guided Multimodal Alignment and Correction module that leverages text as the dominant modality to guide the correction of auxiliary modalities (audio and visual), reducing the semantic gap between them. This correction process is further enhanced by a cross-modal alignment mechanism, ensuring that conflicting information across modalities is resolved effectively. Additionally, CMSA incorporates Context-aware mechanisms to adjust the model’s behavior based on contextual variations by utilizing Context-aware Mixture-of-Experts (CMoE), optimizing performance across diverse scenarios. Experimental results on several benchmark datasets demonstrate the superiority of our approach. CMSA outperforms state-of-the-art methods in terms of both sentiment classification accuracy and robustness to modality conflicts, confirming its effectiveness in multimodal sentiment analysis tasks. The proposed model’s ability to adapt to different contexts and dynamically refine modality contributions presents a promising direction for future research in multimodal emotion recognition and sentiment analysis.