Improving Emotion Recognition with Pre-Trained Models, Multimodality, and Contextual Information

With the widespread adoption of internet technology and the rapid development of information technology, an increasing number of viewpoints are being conveyed on online platforms, making emotion recognition of these viewpoints very important. Information is typically conveyed in multimodal forms, such as text, audio, and video modalities. Combining information from these modalities can enhance emotion recognition. This study integrates information from two modalities: the BERT model and the RoBERTa model are introduced for text modality to extract features, and the data2vec2.0 model is introduced for audio modality to extract features, incorporating contextual information from previous dialogues for training. All experiments in this paper are conducted on the IEMOCAP dataset, using a five-fold cross-validation method. The final multimodal emotion recognition achieves a weighted accuracy of 78.49%, an unweighted accuracy of 79.38%, and an F1-score of 78.43%. The models and code can be found on https://github.com/xiazhengshun/multimodal-emotion-recognition.