Audio-Sensitive Speech Emotion Recognition via Content- Independent Pretraining and Threshold-Based Fusion

Speech emotion recognition (SER) has attracted increasing attention in human–computer interaction, mental health monitoring, and multimedia retrieval. However, many existing multimodal SER systems exhibit a strong bias toward the text modality: because utterance-level labels are often easily inferred from lexical content, models tend to under-utilize non-verbal acoustic cues, which can lead to erroneous predictions when crucial emotional information is predominantly conveyed by prosodic and spectral features. To alleviate this imbalance, we propose an audio-sensitive SER framework that explicitly enhances the contribution of the audio modality through a two-step strategy. First, we construct an Audio Sensitive Network (ASN) by pretraining on the parallel Emotional Speech Dataset (ESD), in which identical linguistic content is spoken with different emotions. This setting allows the ASN to learn speech content-independent emotional representations that emphasize paralinguistic information. Second, we introduce a threshold fusion scheme that integrates the ASN with existing SER classifiers. Specifically, we employ the Tree-structured Parzen Estimator (TPE) to optimize label-wise decision thresholds, enabling flexible calibration of the joint prediction space across modalities and models. We conduct experiments on both the IEMOCAP and ESD corpora, comparing multiple baseline classifiers with and without the proposed audio-sensitive enhancement. The results show consistent, albeit moderate, improvements in emotion recognition performance (e.g., up to +11.7% absolute accuracy on angry for MMAN on IEMOCAP), particularly for emotions that rely heavily on prosodic and spectral cues, thereby demonstrating the effectiveness of the proposed framework in boosting audio sensitivity within multimodal SER systems.