Addressing Emotion Ambiguity and Annotator Subjectivity for Enhanced Speech Emotion Labeling

Conventional hard-label and soft-label labeling strategies for Speech Emotion Recognition (SER) fail to capture the diversities in annotator expertise in perceiving complex human emotions. This study introduces novel soft-label approaches that integrate emotion-specific annotator abilities and optimize database utilization by incorporating non-consensus and majority-voted non-target class utterances. We evaluate our methods primarily on the IEMOCAP database, with additional verification using the MSP-Podcast database, employing features from wav2vec 2.0 and WavLM models. Compared to the conventional hard-label approach, our method improves Unweighted Accuracy (UWA) by 6.80% and 2.73% with WavLM features for the IEMOCAP and MSP-Podcast databases, respectively. The results also outperform other state-of-the-art soft-label approaches in SER literature. Our study shows the importance of accounting for the annotator’s expertise and the inherent subjectivity in emotional perception for improving the SER performance.