Audio-Visual Learning for Multimodal Emotion Recognition

Most current emotion recognition methods are often limited to a single- or dual-modality approach, neglecting the rich information embedded in other modalities. This limitation hampers the accurate identification of complex or subtle emotional expressions. Additionally, to reduce the computational cost during inference, minimizing the model’s parameter size is essential. To address these challenges, we utilize the concept of symmetry to design a balanced multimodal architecture that integrates facial expressions, speech, and body posture information, aiming to enhance both recognition performance and computational efficiency. By leveraging the E-Branchformer network and using the F1- score as the primary performance evaluation metric, the experiments are mainly conducted on the CREMA-D corpora. The experimental results demonstrate that the proposed model outperforms baseline models on the CREMA-D dataset and an extended dataset incorporating eNTERFACE’05, achieving significant performance improvements while reducing the number of parameters. These findings demonstrate the effectiveness of the proposed approach and provide a new technical solution for the field of emotion recognition.