Dynamic Mutual Refinement for Multimodal Speech Emotion Recognition: A Bidirectional Distillation and Inverse Curriculum Learning Approach

With the rapid advancement of speech emotion recognition, the transition from unimodal to multimodal approaches has become inevitable. However, multimodal methods introduce new challenges, particularly classification ambiguity, where multimodal models may misclassify samples that unimodal models correctly identify, and vice versa. To address this, we propose Dynamic Adaptive Mutual Refinement (DAMR), a training framework designed to reduce classification ambiguity between multimodal and unimodal models while preserving their core architectures. DAMR incorporates three key components: (1) dynamic mutual distillation (DMD), which employs an EMA-based capability assessment mechanism and competitive dynamic weighting to adaptively adjust knowledge transfer between models; (2) adaptive inverse curriculum learning (AICL), which implements a hard-to-easy learning schedule based on information bottleneck theory and gradient signal analysis, focusing on informative and difficult samples in early training stages; and (3) an integrated optimization framework that jointly applies DMD and AICL through adaptive weighting. Experiments on the MELD and IEMOCAP datasets demonstrate that our approach outperforms state-of-the-art methods in emotion recognition, achieving a notable 5.54% improvement over the baseline on IEMOCAP.