LoRA-Efficient Multi-Task Learning with Joint Semantic and Paralinguistic Distillation for Speech Emotion Recognition

Speech Emotion Recognition (SER) is an important but challenging research topic in affective computing, aiming to identify the emotional state of a speaker from their speech. Emotional information in an utterance is carried by two complementary channels: the semantic content of the words (what is said) and the paralinguistic manner of delivery (how it is said). Self-supervised speech models such as WavLM capture acoustic patterns well, but they do not explicitly model lexical-semantic emotional cues. Multimodal alternatives, on the other hand, depend on automatic speech recognition (ASR) transcripts at inference time, increasing the deployment cost. To address these issues, we propose a LoRA-efficient multi-task learning framework built on a WavLM backbone, in which the semantic and paralinguistic representations of emotional speech are learned jointly inside a single speech encoder. The framework trains three objectives together: semantic distillation, paralinguistic self-reconstruction, and emotion classification, where the latter is treated as the primary task and the other two provide supporting supervision. Experiments on MSP-IMPROV and IEMOCAP validate the effectiveness of the proposed framework, and on IEMOCAP the model outperforms several existing SER methods, confirming the rationality of the design.