Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks

Most studies indicate that bimodal emotion recognition systems are more objective and accurate. However, many of these systems depend on physiological signals that require direct measurement, which introduces certain limitations. This study aims to develop a new bimodal emotion recognition system based on skin temperature (SKT) and heart rate variability (HRV) using hybrid neural networks. Notably, these physiological signals can be measured remotely, addressing the limitations of direct measurement methods. The integration of these modalities enables the model to effectively utilize both spatial and temporal features for robust emotion classification. The hybrid neural network, combining a convolutional neural network and a gated recurrent unit (CNN+GRU), was trained on experimental SKT and HRV data collected from individuals experiencing basic emotions such as anger, disgust, fear, happiness, sadness, and surprise. The trained model achieved an accuracy of 95.58%, outperforming existing approaches that use only a single data modality. Confusion matrix analysis demonstrated high accuracy in recognizing most basic emotions. The results confirm the effectiveness of the proposed approach in combining physiological and visual signals for improved emotion recognition.