GMHCA-MCBILSTM: A Gated Multi-Head Cross-Modal Attention-Based Network for Emotion Recognition Using Multi-Physiological Signals

To address the limitations of the single-modal electroencephalogram (EEG), such as its single physiological dimension, weak anti-interference ability, and inability to fully reflect emotional states, this paper proposes a gated multi-head cross-attention module (GMHCA) for multimodal fusion of EEG, electrooculography (EOG),and electrodermal activity (EDA). This attention module employs three independent and parallel attention computation units to assign independent attention weights to different feature subsets across modalities. Combined with a modality complementarity metric, the gating mechanism suppresses redundant heads and enhances the information transmission of key heads. Through multi-head concatenation, cross-modal interaction results from different perspectives are fused. For the backbone network, a multi-scale convolution and bidirectional long short-term memory network (MC-BiLSTM) is designed for feature extraction, tailored to the characteristics of each modality. Experiments show that this method, which primarily fuses eight-channel EEG with peripheral physiological signals, achieves an emotion recognition accuracy of 89.45%, a 7.68% improvement over single-modal EEG. In addition, in cross-subject experiments conducted on the SEED-IV dataset, the EEG+EOG modality achieved a classification accuracy of 92.73%. All were significantly better than the baseline method. This fully demonstrates the effectiveness of the innovative GMHCA module architecture and MC-BiLSTM feature extraction network proposed in this paper for multimodal fusion methods. Through the novel attention gating mechanism, higher recognition accuracy is achieved while significantly reducing the number of EEG channels, providing new ideas and approaches based on attention mechanisms and gated fusion for multimodal emotion recognition in resource-constrained environments.