A consistency-constrained fusion network for audio-visual emotion recognition

Audio-Visual Emotion Recognition (AVER) from speech and facial expression is a fundamental task in affective computing, yet effectively fusing multimodal features while accounting for both modality consistency and discrepancy remains a challenging problem. Existing approaches either neglect modality discrepancy or model consistency and discrepancy in separate feature spaces. This paper proposes a Consistency-Constrained Fusion Network (CCFNet) that jointly captures unimodal consistency and discrepancy within a unified architecture. To extract modality consistency, two complementary feature fusion models are designed: an attention-based model that performs unimodal self-fusion followed by multimodal cross-fusion, and a Feature-wise Linear Modulation (FiLM) based model that uses facial expression features to modulate speech channel features. To retain modality discrepancy without performance degradation, a consistency encoder constrained by a Central Moment Discrepancy (CMD) loss maps unimodal features into a shared vector space, allowing discrepant features to be preserved rather than discarded as noise. A gate-structure based model aggregates segment-level fused features into video-level representations, and a two-stage training strategy addresses the different learning dynamics between modalities. Experiments on the RAVDESS and CMU-MOSEI datasets demonstrate the effectiveness of the proposed method, achieving recognition accuracies of 79.03% and 68.27% (weighted accuracy), respectively. Ablation studies confirm that each component of the framework contributes to the overall performance.

Authors

Institutions

Publication Details

Journal
Journal of King Saud University - Computer and Information Sciences
Published
2026-11-19
DOI
https://doi.org/10.1007/s44443-026-01434-7
Primary Topic
Emotion and Mood Recognition
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A consistency-constrained fusion network for audio-visual emotion recognition

Xiang Sui, Zhipeng Liu, Xingcan Liang, Liping Zhu et al.
Journal of King Saud University - Computer and Information Sciences
Emotion and Mood Recognition
article

A consistency-constrained fusion network for audio-visual emotion recognition

Xiang Sui, Zhipeng Liu, Xingcan Liang, Liping Zhu, Linsen Xu
article en

Abstract

Audio-Visual Emotion Recognition (AVER) from speech and facial expression is a fundamental task in affective computing, yet effectively fusing multimodal features while accounting for both modality consistency and discrepancy remains a challenging problem. Existing approaches either neglect modality discrepancy or model consistency and discrepancy in separate feature spaces. This paper proposes a Consistency-Constrained Fusion Network (CCFNet) that jointly captures unimodal consistency and discrepancy within a unified architecture. To extract modality consistency, two complementary feature fusion models are designed: an attention-based model that performs unimodal self-fusion followed by multimodal cross-fusion, and a Feature-wise Linear Modulation (FiLM) based model that uses facial expression features to modulate speech channel features. To retain modality discrepancy without performance degradation, a consistency encoder constrained by a Central Moment Discrepancy (CMD) loss maps unimodal features into a shared vector space, allowing discrepant features to be preserved rather than discarded as noise. A gate-structure based model aggregates segment-level fused features into video-level representations, and a two-stage training strategy addresses the different learning dynamics between modalities. Experiments on the RAVDESS and CMU-MOSEI datasets demonstrate the effectiveness of the proposed method, achieving recognition accuracies of 79.03% and 68.27% (weighted accuracy), respectively. Ablation studies confirm that each component of the framework contributes to the overall performance.

Journal of King Saud University - Computer and Information SciencesVol. 38(10)
University of Science and Technology of China (CN), Hohai University (CN), Jiangsu University of Science and Technology (CN), Changzhou University of Information Technology (CN), Suzhou Institute of Technology, Jiangsu University of Science and Technology (CN)
Emotion and Mood Recognition
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon