Enhancing Multimodal Sentiment Prediction with Cross-Modal Attention and Adaptive Feature Weighting

This study introduces a novel framework, Contextual Adaptive Cross-Modal Attention Fusion (CA-CMAF), designed to solve multimodal sentiment analysis’s difficulties. The framework leverages dynamic modality fusion and cross-modal attention mechanisms to effectually integrate textual and visual data, enabling a more nuanced understanding of sentiment in heterogeneous datasets. By focusing on the interplay between modalities, CA-CMAF aims to increase the accuracy and interpretability of the sentiment prediction system. The proposed approach combines textual features extracted through BERT, which has been both pre-trained and subsequently fine-tuned, with visual features derived from VGG-16. Through a cross-modal attention technique, these modalities are fused and aligned, capturing fine-grained interactions between text and images. The attention mechanism computes attention scores to prioritize the most relevant aspects of each modality, depending on the context provided by others. Additionally, an adaptive learning mechanism dynamically adjusts the contribution of each modality, ensuring optimal fusion for sentiment classification. The model is optimized using a blended loss approach. One part of this approach is based on cross-entropy principles for sentiment classification. Another component includes a regularization term to ensure balanced modality contributions. Experimental findings on MVSA-Single and MVSA-Multiple benchmark datasets indicate that CA-CMAF surpasses current baseline methods in state-of-the-art performance. The framework shows significant boosts in performance for metrics like accuracy, F1-score, precision, recall, with readings of 91%, 89%, 90%, and 90%, respectively, particularly in scenarios where one modality is more informative than the other.