Enhanced multi-modal emotion recognition using the feature level fusion

Multi-modal human emotion recognition is a complex process of synthesizing information from various modalities to calculate emotion states. This field faces several challenges: (1) Acoustic is an essential component of emotion expression, but it often underperforms compared to visual and text in emotion recognition. (2) Capturing the feature interaction among different modalities is usually complex. (3) Processing high-definition videos can significantly reduce the efficiency of visual analysis. In this study, we presented a learning architecture designed to recognize human emotions effectively. For the first challenge, we implemented a multi-level acoustic encoder (MLAE) that enhances the extraction of acoustic information to improve the acoustic contribution in multi-modal emotion recognition. Facing the second challenge, we introduced the cross-attention block module, which adeptly captures the inter-modal interactions. To address the third challenge, we adopted the re-parameterized visual geometry group network (RepVGG) as the visual feature encoder, employing its multi-branch learning and single-branch reasoning structure to maintain high reasoning efficiency. Our model has demonstrated the state-of-the-art performance of the interactive emotional dyadic motion capture (IEMOCAP) dataset and the multi-modal opinion sentiment and emotion intensity of the Carnegie Mellon University (CMU-MOSEI) dataset.