COLIN: complementary and competitive balanced learning network for multi-modal multi-label emotion recognition

Multi-modal multi-label emotion recognition (MMER) aims to identify multiple co-occurring emotions from heterogeneous modalities. Existing methods tend to emphasize either modality complementarity or modality-specific information, while rarely achieving a balanced integration of both. Moreover, dependencies between modalities and emotion labels remain insufficiently explored, and the challenge of label imbalance in MMER has received little attention. To address these challenges, we propose COLIN (COmplementary and competitive baLanced learnIng Network), a unified framework designed to simultaneously balance modality complementarity and specificity while alleviating learning imbalance. Specifically, COLIN incorporates a Mixture-of-Experts (MoE) mechanism with attention to capture label dependencies and optimize multi-modal feature representations. It further adopts a dual-branch architecture that integrates modality-specific feature reconstruction with multi-level fusion, ensuring balanced utilization of information across different modalities. In addition, an improved loss function is designed to mitigate label imbalance and enhance overall model robustness. Extensive experiments on two benchmark datasets, CMU-MOSEI and $$\text {M}^3\text {ED}$$, demonstrate that COLIN consistently outperforms state-of-the-art methods, validating its effectiveness in both modality balancing and robust multi-label learning.