Cross-modal Emotion-specific Attention model for Multimodal Emotion Recognition

Emotion recognition plays a crucial role in humanrobot interaction, where accurately interpreting human emotions through multiple modalities is essential for heartfelt communication. Although previous multimodal emotion recognition models have shown reasonable performance, there are two difficulties: (1) they often struggle to capture the fine-grained interactions between the modalities. (2) The prominent features are different across different emotions. To address these difficulties, we propose a novel Cross-modal Emotion-specific Attention model (CEA) for multimodal emotion recognition. The proposed model has two key components to address the difficulties above: (1) To capture the fine-grained interactions, we introduce the dense interaction matrix representation. (2) To focus more on emotion-specific features, we also introduce the specific emotion tokens. Combining these two components enhances the model’s ability to capture subtle emotional nuances and improves overall recognition accuracy. We evaluate the performance of the proposed architecture on the CREMA-D public audiovisual datasets through comprehensive ablation studies and comparison with baseline models. The results demonstrate that our Cross-modal Emotion-specific Attention model significantly outperforms the baseline methods, confirming its effectiveness in enhancing emotion recognition accuracy.