Dual-channel cross-attention fusion of heterogeneous facial and physiological signals for emotion recognition
Effective fusion of heterogeneous multimodal data is challenging when modalities differ substantially in structure, temporal resolution, and representation space. In emotion recognition, facial behavior and physiological signals provide complementary information but exhibit precisely these differences. We propose an end-to-end framework with a Dual-channel Multimodal Interaction Module (DMIM) for bidirectional fusion of facial and physiological representations. A pretrained convolutional neural network extracts facial features, while a Temporal-Spatial CNN models temporal and inter-channel dependencies in physiological signals. DMIM employs two cross-attention paths that allow each modality to query the other without requiring matched sequence lengths, followed by learnable linear combinations and soft feature selection. Experiments were conducted on the DEAP dataset using the 22 participants with both visual and physiological recordings under random-segment validation and additional subject-independent 10-fold testing. The original random-segment analysis achieved best-validation accuracies of 99.58% for valence and 99.59% for arousal. Within the original validation protocol, removing cross-attention reduced both accuracies by 0.26 percentage points, while multimodal fusion outperformed either modality alone. Additional subject-independent testing yielded accuracies of $$50.79\% \pm 5.05\%$$ for valence and $$50.65\% \pm 8.16\%$$ for arousal. The additional matched ablations did not establish a consistent Full-model advantage on unseen participants; the findings therefore delimit the performance of the proposed architecture under the evaluated protocols.
Authors
- Xuewen Wang (ORCID: https://orcid.org/0009-0006-6244-5453)
- Daojun Gong (ORCID: https://orcid.org/0000-0003-3337-6146)
- Chenxi Cao
- Yimin Wang
- Lei Luo
Institutions
- Macau University of Science and Technology (MO)
- Northwestern Polytechnical University (CN)
- Communication University of China (CN)
Publication Details
- Journal
- Journal Of Big Data
- Published
- 2026-11-21
- DOI
- https://doi.org/10.1186/s40537-026-01609-z
- Primary Topic
- Emotion and Mood Recognition
- Type
- article
- Field-Weighted Citation Impact
- 0.00