Multimodal Emotion Recognition with Balanced Audio and Video Feature Extraction Under Text Modality Residuals

With the rise of social media and digital communication, sentiment analysis has become an important part of artificial intelligence. In this paper, we study the problems of insufficient inter-modal correlation and inefficient feature extraction in multimodal sentiment recognition. Based on the self-MM model and combining with the attention mechanism, we propose a model named VAMM-FQM, which deepens the correlation between the two modalities through the self-attention mechanism and improves the model efficiency through the feedback mechanism. The performance of this model is improved by about 5 % over the original model, and the correlation is improved by more than two times. Next, by optimising the feature extraction part and replacing the sLSTM method with EfficientNet and Wav2vec 2.0 for the visual and audio parts of the feature extraction, respectively, a model called VAMM-WE is proposed, which ensures that detailed features of both domains can be captured. In addition, Transformer is used as a decision-level fusion operator to facilitate information exchange between the two modalities, achieving a significant error rate reduction. Compared to the original model, the new model reduced the error rate by 60 % on each dataset.