Emotion Recognition Model Using Progressive Transfer Learning for Speech Data Adaptation

Due to the increase in remote communication caused by the COVID-19 pandemic, the importance of emotion recognition in education and psychological counseling has becoming more prominent. Accordingly, multimodal approaches have recently received attention, and research on emotion recognition using these approaches is actively underway. The multimodal approach uses ensemble techniques that average or weight model predictions to predict the final result. However, if one modality shows relatively low performance, it can limit the performance of the entire model. Therefore, to improve the performance of multimodal emotion recognition, it is necessary to enhance the performance of individual modalities. In this paper, we propose a speech data adaptive emotion recognition model using progressive transfer learning by applying domain adaptation techniques. After initially training the designed deep learning model with text data, the fixed weights are progressively reduced and retrained with voice data to effectively combine the characteristics of text and voice for speech emotion recognition. Experimental results confirm that this approach achieves higher performance than models trained solely on voice data and models using general transfer learning. This methodology is expected to contribute to improving multimodal emotion recognition performance and enhancing interaction in non-face-to-face environments.