Cross-Attention Dual-Stream Fusion for Speech Emotion Recognition

Speech emotion recognition aims for recognizing human subjective emotions through in-depth audio signal analysis. It benefits a wide range of downstream applications and tasks. However, how to comprehensively encode emotion cues and how to effectively fuse implicit information still remains challenging. To address the issues, dual-stream representation is designed that uses full-training of TextRCNN from log-Mel spectrograms and fine-tuning of pre-trained HuBERT from raw audio signals for emotion encoding, and cross-attention fusion (CAF) is developed for valid integration of dual-stream outputs. On the IEMOCAP of four emotions (“happy+excited”, “sad”, “angry”, and “neutral”), the effectiveness of the framework is evaluated by using different dual-stream encoders (DPCNN and pre-trained Wave2Vec2) and different fusion modules (summation, concatenation, feature-wise linear modulation, and CAF). Experimental results reveal that the proposed framework achieves promising performance (weighted accuracy, 0.6965; unweighted accuracy, 0.7088; F1-score, 0.7084), the dual-stream encoder enriches emotion representation, and the CAF outperforms the other three modules. For further improving the performance, multi-stream representation and multi-branch feature fusion would be the research direction. Our code will be available on GitHub (https://github.com/NicoYuCN/serCAF).