A Study on Multimodal Fusion and Layer Adapter in Emotion Recognition

Multimodal emotion recognition (MER) is a rapidly evolving field aimed at integrating information from various modalities, such as speech and text, to deepen our understanding of emotions. However, challenges in feature extraction and fusion hinder further advances in MER performance. To address these challenges, we propose different modality representations to capture emotional information comprehensively. Additionally, we introduce a novel layer adapter and multimodal fusion method to explore modality-dependent and modality-invariant interactions in MER. Extensive experimental results show the effectiveness of our approach, achieving state-of-the-art results with an absolute improvement of 1.89% over the baseline. The significant improvements validate the effectiveness of our proposed method in enhancing the MER system.