Multi-Task Learning-Based Speech Emotion Recognition Using Pre-Trained Acoustic Model

Accurate recognition of human emotions is crucial for human–computer interaction, and speech, as an important external manifestation of emotion, has attracted significant attention. Existing speech emotion recognition (SER) methods are predominantly based on single-task learning, which inadequately model speaker variability and other latent factors in speech, thereby limiting recognition performance. In this paper, a multi-task learning-based SER method leveraging a pre-trained acoustic model is proposed. Speech emotion recognition is treated as the primary task, while speaker recognition, gender recognition, and automatic speech recognition are introduced as auxiliary tasks. A multi-task learning framework based on hard parameter sharing is constructed to guide the model to learn shared acoustic representations that simultaneously encode emotional category characteristics, speaker identity, and other relevant information. Experiments conducted on the IEMOCAP dataset demonstrate that the proposed model achieves weighted accuracy (WA) and unweighted accuracy (UA) of 83.24% and 83.36%, respectively, under five-fold cross-validation, and 83.86% and 84.23%, respectively, under ten-fold cross-validation. In both settings, the proposed method consistently outperforms the baseline models, confirming its effectiveness in improving speech emotion recognition performance.