Multimodal emotion recognition based on feature aggregation and multi-objective optimization

In order to extract deep emotional features across multiple modalities and to mitigate high computation complexity and information redundancy in intermodal interaction to improve accuracy, we propose a multimodal emotion recognition model designed for feature aggregation and multi-objective optimization. Two encoders pre-trained with wav2vec-2.0 and RoBERTa-base extract features from speech and text signals, respectively, with a custom aggregation block fusion for intermodal interaction. Additionally, it addresses three auxiliary speech emotion recognition tasks as multi-objective optimization. Finally, we evaluate our proposed method on two corpora, IEMOCAP and CMU-MOSEI. Empirical results demonstrate that our proposed method performed better on all benchmarks compared to existing state-of-the-art. The proposed method is established.