Robust Cross-Corpus Speech Emotion Recognition without Emotion Labels in Pretraining

Speech Emotion Recognition (SER) has gained increasing attention in recent years, yet its progress has been limited due to the scarcity and low diversity of emotion-labeled datasets. This study addresses this challenge by investigating whether pretraining on a large-scale speech dataset without emotion labels can enhance the performance and robustness of SER models, especially under low-resource conditions. Using the LibriTTS-P dataset, we conducted pretraining on acoustic feature classification tasks—pitch, energy, and speaker gender—and subsequently fine-tuned the pretrained models using three small-scale emotional speech datasets (RAVDESS, SAVEE, and TESS). To assess cross-dataset generalization, we evaluated each model on unseen datasets. The results show that models pretrained on more complex acoustic tasks achieved higher accuracy and better generalization than those trained from scratch. These findings suggest that pretraining on large and diverse speech data without emotion labels, combined with well-designed tasks, can effectively reduce the dependence on scarce emotion-labeled data and improve the practical robustness of SER systems.