Exploiting Input Space Translation and Contrastive Learning for Multimodal Sentiment Analysis

Multimodal sentiment analysis is grounded in textual data and incorporates speech and visual features to accurately analyze sentiment. However, the diversity exhibited by social media twitters, some of which contain minimal textual content, as well as instances of complex visual information or images that contradict the accompanying text, pose challenges in extracting effective emotional information. We propose a model that combines Contrastive Learning and Multimodal Fusion Method for multimodal sentiment analysis tasks. Specifically, we initially encode text and image inputs. Additionally, we employed a caption transformer encoder to generate image captions and designed two contrastive learning tasks utilizing captions. We conduct extensive experiments on multimodal datasets to demonstrate the effectiveness of our proposed method.