EXPLORING AND ANALYZING MULTIMODAL EMOTION ANALYSIS TASKS UNDER DIFFERENT MODAL FUSION METHODS

The objective of this paper is to investigate the effectiveness of different modality fusion methods on multimodal sentiment analysis. Specifically, the task is based on image-text sentiment analysis using the Multi-View Sentiment Analysis (MVSA) Twitter dataset. In this paper, pre-trained BERT and ResNet-50 are used as feature extractors for text and images, respectively. The author constructed different feature fusion models based on different feature fusion methods. The different models were compared and analyzed for accuracy, F1-score, and confusion matrix of the sentiment analysis task. At the same time, ablation experiments were conducted to prove the effectiveness of modality fusion, and some typical errors that occurred during the experiment and their causes were analyzed. Finally, the effect of training data size was studied. This paper contributed ideas and references for researchers in the field of multimodal feature fusion.