Emotion Recognition Based on Fusion of Multi-modal Physiological Signals

Emotion recognition based on physiological signals has emerged as a key research focus in affective computing due to its objectivity and robustness. This paper presents a novel framework for emotion recognition that integrates multi-modal physiological signals. The proposed method leverages a convolutional neural network for local feature extraction, employs a multi-modal Transformer to model both intra-modal and inter-modal interactions, and incorporates a contrastive learning strategy to enhance feature alignment and class separability. Experiments conducted on the WESAD dataset demonstrate that the proposed framework outperforms both traditional machine learning and existing deep learning baselines, particularly in the fusion of EDA and EMG modalities. A detailed analysis of confusion matrices further confirms the model’s capability in reducing inter-class misclassification. These results underscore the potential of the proposed approach for building robust and generalizable emotion recognition systems applicable to real-world scenarios.