A Transfer Learning-Based Text-Centric Model for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MMSA) is a research method that extracts effective information from heterogeneous modal information. Then, MMSA processes the multimodal data and performs sentiment analysis. Along with big data and machine learning development, multimodal sentiment analysis has become a hot research direction in multimodal learning and natural language processing. Although various feature extraction methods and information fusion methods have been continuously proposed, challenges exist in MMSA research. First, in terms of feature extraction, pre-trained models trained with many data sets can obtain higher quality features, but research on how to use these feature extraction methods to extract the best features is still needed. Currently, the more popular feature fusion methods do not focus on the interaction between multiple modal information and the retention of basic information. To overcome these problems a multimodal sentiment analysis model utilizes text features as core modal features, using video and audio modal features as auxiliary modal features, multimodal feature modality attention mechanism to extract the intrinsic connection between different modalities. The attention mechanism uses the features of video modality and audio modality as the focus and then enhances the text modality with the fusion of video modality and modality. To improve the quality of extracted features, this method chooses the transfer learning training method and uses the pre-trained model for processing. This research uses the CMU-MOSI dataset to test the proposed method. Experimental results show that the performance of the proposed model in emotion score prediction and emotion classification tasks exceeds traditional methods and baseline methods.