Multimodal Physiological Signal Emotion Recognition Method Based on Attention Mechanism

Prior research in emotion recognition has mainly concentrated on examining physiological signals from a single modality, such as electroencephalogram (EEG) or various peripheral physiological signals (PPS). However, given the multimodal nature of human emotional expression, these unimodal methodologies fail to reach satisfactory accuracy. To overcome this challenge, we introduce a multimodal emotion recognition approach utilizing the attention mechanism (MERA), consisting of both a unimodal feature extraction module and a multimodal feature fusion module. During the unimodal feature extraction phase, we investigate the complex interactions among various channels and employ an encoder module to precisely capture the subtle distinctions in emotional fluctuations. In the multimodal feature fusion phase, a multi-head attention mechanism is employed to emphasize the complementary nature of various modalities and the complex interplay among signals through dynamic weighting, thereby enhancing the accuracy of emotion recognition. Experiments carried out on the DEAP and MAHNOB-HCI datasets reveal superior outcomes for our method in binary and four-class emotion classification tasks across various frequency bands. Our framework outperforms contemporary leading approaches in terms of accuracy for all types of classification.