A consistency-constrained fusion network for audio-visual emotion recognition
Audio-Visual Emotion Recognition (AVER) from speech and facial expression is a fundamental task in affective computing, yet effectively fusing multimodal features while accounting for both modality consistency and discrepancy remains a challenging problem. Existing approaches either neglect modality discrepancy or model consistency and discrepancy in separate feature spaces. This paper proposes a Consistency-Constrained Fusion Network (CCFNet) that jointly captures unimodal consistency and discrepancy within a unified architecture. To extract modality consistency, two complementary feature fusion models are designed: an attention-based model that performs unimodal self-fusion followed by multimodal cross-fusion, and a Feature-wise Linear Modulation (FiLM) based model that uses facial expression features to modulate speech channel features. To retain modality discrepancy without performance degradation, a consistency encoder constrained by a Central Moment Discrepancy (CMD) loss maps unimodal features into a shared vector space, allowing discrepant features to be preserved rather than discarded as noise. A gate-structure based model aggregates segment-level fused features into video-level representations, and a two-stage training strategy addresses the different learning dynamics between modalities. Experiments on the RAVDESS and CMU-MOSEI datasets demonstrate the effectiveness of the proposed method, achieving recognition accuracies of 79.03% and 68.27% (weighted accuracy), respectively. Ablation studies confirm that each component of the framework contributes to the overall performance.
Authors
- Xiang Sui (ORCID: https://orcid.org/0009-0009-4173-7520)
- Zhipeng Liu (ORCID: https://orcid.org/0000-0001-8436-4925)
- Xingcan Liang (ORCID: https://orcid.org/0000-0002-3407-2905)
- Liping Zhu
- Linsen Xu
Institutions
- University of Science and Technology of China (CN)
- Hohai University (CN)
- Jiangsu University of Science and Technology (CN)
- Changzhou University of Information Technology (CN)
- Suzhou Institute of Technology, Jiangsu University of Science and Technology (CN)
Publication Details
- Journal
- Journal of King Saud University - Computer and Information Sciences
- Published
- 2026-11-19
- DOI
- https://doi.org/10.1007/s44443-026-01434-7
- Primary Topic
- Emotion and Mood Recognition
- Type
- article
- Field-Weighted Citation Impact
- 0.00