Multimodal sentiment analysis and fusion based on deep learning

To address the issues of low accuracy and limited generalization in existing multimodal emotion recognition methods, particularly in feature extraction and cross-modal integration, a novel algorithm integrating speech, text, and image data is introduced. Firstly, multiple Long Short-Term Memory Networks were used to extract features from image modalities, while WavLM and RoBERTa models were used to extract features from speech and text modes, respectively. Secondly, in order to improve the effect of modal fusion, the self-attention Transformer layer was used to fuse the speech and text modalities, and the time-position prediction method was used to achieve the alignment between the modalities. Finally, in the final learning recognition stage, a contrastive learning strategy is introduced to distinguish the eigenvalues of different emotions more effectively by narrowing the Euclidean distance between positive samples and expanding the distance between negative samples. The proposed model, tested on the CMU-MOSI dataset, outperforms other methods, achieving recognition rates of 72.42% for three-class classification and 78.65% for binary classification, validating its efficacy.