Multimodal emotion recognition based on audiovisual text

The article constructs a hybrid fusion model for multimodal emotion recognition. First, facial expression features are extracted using Mediapipe and OpenCV, and then a convolutional neural network, bi-directional long and short-term memory combined with SelfAttention mechanism network model (CNN-BiLSTM - SelfAttention) is used to train for the expression features; the speech part of the paper uses an LSTM model to train the audio features; and the text part of the paper extracts Chinese text features with the help of the Roberta model. Roberta model is used to extract the Chinese text features, and then the two-way long and short-term memory network combined with the self-attention mechanism (BiLSTM - SelfAttention) is used to train the text features; based on the above operations, the self-attention mechanism, the multi-attention mechanism and the cross-attention mechanism combined with the BiLSTM and Transformer are used for the feature-level fusion and the hybridization of features, respectively. The three models, text, expression and audio, are fused together in a feature-level fusion amount and hybrid fusion, respectively. In this way, the models are able to learn the features of the three modalities, which in turn improves the ability to predict emotion categories. Experimental validation was carried out on the CH - SIMS public dataset, and the experimental results showed that the accuracy after expression, speech, text fusion, and multimodal fusion reached 86.08%, 83.61%, 91.29%, and 91.49%, respectively.