A Tiny Whisper-SER: Unifying Automatic Speech Recognition and Multi-label Speech Emotion Recognition Tasks

Speech Emotion Recognition (SER) is critical in human-computer interaction (HCI). Previous SER works often utilize Automatic Speech Recognition (ASR) systems to improve SER performance. Some studies have investigated the possibility of modeling multi-task ASR and SER. However, prior studies and our preliminary experiments show the conflicts between ASR and SER if standard multi-task learning is used. We adopt a two-stage and weighted training strategy to overcome the conflicts between SER and ASR tasks. We utilize the public Whisper model that supports ASR tasks and others and add a small light adapter on the Whisper for the SER task, Whisper-SER. We first fine-tune the model on the ASR task and then fine-tune the model using the weighted loss, controlling loss between the ASR and multi-label SER task during training. The proposed method allows the Whisper-SER to recognize emotions and transcribe without ASR and SER performance degradation within the same encoder and decoder.