Multi-Label Emotion Share Regression from Speech Using Pre-Trained Self-Supervised Learning Models

Speech emotion recognition (SER) has been enabling natural human-machine interaction. Previous SER only focused on predicting a single emotion from humans' voices, whereas emotions from voices might have multiple categories. This study aims to predict emotion share from a voice, i.e., the proportion of the different emotion categories expressed in the speech. We evaluated eight pretrained self-supervised learning (SSL) models from major companies for multilingual emotion share recognition. The models were evaluated on the Hume-Prosody dataset, which contains 51,881 samples of nine different emotion categories. We observed the important factor of wider window size for multilingual emotion share recognition in contrast to the narrow window size used (around 5.5 s). The most effective evaluated pretrained model for this case is the large version of HuBERT, which achieved a Spearman correlation score of 0.579, showing the feasibility of emotion share recognition using SSL.