Dual-channel cross-attention fusion of heterogeneous facial and physiological signals for emotion recognition

Effective fusion of heterogeneous multimodal data is challenging when modalities differ substantially in structure, temporal resolution, and representation space. In emotion recognition, facial behavior and physiological signals provide complementary information but exhibit precisely these differences. We propose an end-to-end framework with a Dual-channel Multimodal Interaction Module (DMIM) for bidirectional fusion of facial and physiological representations. A pretrained convolutional neural network extracts facial features, while a Temporal-Spatial CNN models temporal and inter-channel dependencies in physiological signals. DMIM employs two cross-attention paths that allow each modality to query the other without requiring matched sequence lengths, followed by learnable linear combinations and soft feature selection. Experiments were conducted on the DEAP dataset using the 22 participants with both visual and physiological recordings under random-segment validation and additional subject-independent 10-fold testing. The original random-segment analysis achieved best-validation accuracies of 99.58% for valence and 99.59% for arousal. Within the original validation protocol, removing cross-attention reduced both accuracies by 0.26 percentage points, while multimodal fusion outperformed either modality alone. Additional subject-independent testing yielded accuracies of $$50.79\% \pm 5.05\%$$ for valence and $$50.65\% \pm 8.16\%$$ for arousal. The additional matched ablations did not establish a consistent Full-model advantage on unseen participants; the findings therefore delimit the performance of the proposed architecture under the evaluated protocols.

Authors

Institutions

Publication Details

Journal
Journal Of Big Data
Published
2026-11-21
DOI
https://doi.org/10.1186/s40537-026-01609-z
Primary Topic
Emotion and Mood Recognition
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Dual-channel cross-attention fusion of heterogeneous facial and physiological signals for emotion recognition

Xuewen Wang, Daojun Gong, Chenxi Cao, Yimin Wang et al.
Journal Of Big Data
Emotion and Mood Recognition
article

Dual-channel cross-attention fusion of heterogeneous facial and physiological signals for emotion recognition

Xuewen Wang, Daojun Gong, Chenxi Cao, Yimin Wang, Lei Luo
article en

Abstract

Effective fusion of heterogeneous multimodal data is challenging when modalities differ substantially in structure, temporal resolution, and representation space. In emotion recognition, facial behavior and physiological signals provide complementary information but exhibit precisely these differences. We propose an end-to-end framework with a Dual-channel Multimodal Interaction Module (DMIM) for bidirectional fusion of facial and physiological representations. A pretrained convolutional neural network extracts facial features, while a Temporal-Spatial CNN models temporal and inter-channel dependencies in physiological signals. DMIM employs two cross-attention paths that allow each modality to query the other without requiring matched sequence lengths, followed by learnable linear combinations and soft feature selection. Experiments were conducted on the DEAP dataset using the 22 participants with both visual and physiological recordings under random-segment validation and additional subject-independent 10-fold testing. The original random-segment analysis achieved best-validation accuracies of 99.58% for valence and 99.59% for arousal. Within the original validation protocol, removing cross-attention reduced both accuracies by 0.26 percentage points, while multimodal fusion outperformed either modality alone. Additional subject-independent testing yielded accuracies of $$50.79\% \pm 5.05\%$$ for valence and $$50.65\% \pm 8.16\%$$ for arousal. The additional matched ablations did not establish a consistent Full-model advantage on unseen participants; the findings therefore delimit the performance of the proposed architecture under the evaluated protocols.

Journal Of Big Data
Macau University of Science and Technology (MO), Northwestern Polytechnical University (CN), Communication University of China (CN)
Emotion and Mood Recognition
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Dual-channel cross-attention fusion of heterogeneous facial and physiological signals for emotion recognition — Xuewen Wang, Daojun Gong, et al. · Journal Of Big Data (2026) | TGRS Research Map | TGRS