MIME: Mutual information-guided mixture of experts for robust multimodal sentiment analysis

Multimodal Sentiment Analysis (MSA) aims to predict human emotions by integrating textual, acoustic, and visual signals. While existing methods have achieved remarkable progress, most approaches treat all modalities equally without considering their inherent reliability differences. In real-world scenarios, acoustic and visual modalities are often corrupted by noise or occlusion, whereas textual signals typically remain semantically reliable. We propose MIME (Mutual Information-guided Mixture of Experts), a novel framework that leverages text as a semantic anchor to guide non-verbal modality learning. MIME introduces modality-specific MoE layers for audio and visual modalities, where multiple lightweight experts offer diverse transformation pathways. Unlike conventional routing based solely on attention or similarity, we propose a hybrid routing mechanism combining content-based matching with MI-guided semantic modulation. The MI factor acts as a semantic filter that suppresses experts whose outputs deviate from the textual anchor while amplifying those with high cross-modal alignment. Furthermore, we design an Early-MI objective that maximizes audio-text and visual-text MI for cross-modal alignment while minimizing audio-visual MI to encourage complementary representations. Experiments on CMU-MOSI and CMU-MOSEI demonstrate that MIME achieves competitive performance and exhibits improved robustness under various noise conditions.