Robust speech emotion recognition under human speech noise

Speech emotion recognition (SER) is the task of predicting human emotions from speech signals. Building robust SER systems under noisy conditions remains challenging due to different noise properties. Most previous studies have overlooked the impact of human speech noise, thereby limiting the applicability of SER. To address this issue, we propose a novel two-stage SER framework that combines target speaker extraction (TSE) with a speaker-aware fusion network (SAFN). In the first stage, we pretrain a TSE model to effectively extract high-quality speech from the target speaker in mixtures. In the second stage, the pretrained TSE is fine-tuned to adapt to emotional contexts while continuing to suppress interfering speakers. Additionally, SAFN integrates the output of TSE with the target speaker’s enrollment information to generate a speaker-aware representation that emphasizes emotion-relevant cues of the target speaker. Comparative experiments demonstrated that our method significantly outperforms a typical SER baseline in the multi-speaker speech noise conditions. • Proposes a two-stage SER framework combining TSE with SAFN. • SAFN and TSE enhance accuracy and robustness under multi-speaker human speech noise. • Investigates SER performance under human speech noise with varying attributes. • Analyzes impacts of emotional attributes and lengths of enrollment on SER performance. • Ablation studies verify the robustness of SAFN under multi-speaker human speech noise.