Multi-task Multimodal Sentiment Analysis Based on Visual Data Mining

In comparison to unimodal data, multimodal data provides a greater range of features, improving the model's effectiveness in sentiment analysis. However, effectively extracting and integrating emotional features from these diverse modalities remains a significant challenge. Heterogeneity and noise present in both visual and textual data frequently result in feature shifts and information loss, which can compromise the stability and precision of sentiment recognition. In response to this challenge, this paper introduces a new multimodal sentiment analysis model referred to as MVDM. First, we extract semantic information from visual elements, including generating image scene caption and recognizing text embedded in the image. Then, we introduce a gating mechanism and a text-driven sparse attention module to denoise and fuse the multimodal features. Finally, we calculate the centroids of different emotion categories by accumulating multiple batches of samples, ensuring accurate alignment of emotional features over a broader range. Additionally, we employ a sliding-label cross-entropy loss strategy to further calibrate feature shift and reduce the risk of model overfitting. The results of Experimental demonstrate that MVDM significantly improves the accuracy and stability of multi-modal sentiment recognition.