Multimodal Emotion Recognition via Correlation of EEG, Thermal, and Digital Image Data

The objectives of this research are (i) to examine EEG signal changes related to various emotional states and relate them to facial thermal and digital image features; (ii) to propose a multimodal deep-learning-based approach for facial emotion recognition (FER) by Combining, EEG, thermal, and digital image modalities; and (iii) to compare the performance of multimodal fusion in enhancing classification accuracy. Fifty participants were exposed to six emotions (happy, sadness, neutral, anger, surprise, and fear) and were collected from to get the data. EEG was pre-processed based on normalization, bandpass filtration (0.5–50 Hz), and segmentation into constant-length epochs. Thermal and digital images were also preprocessed as cropping, resizing, and augmentation of data were performed. Feature extraction was carried out on every modality, where EEG features like Theta Power and Skewness were compared with image-based features like Entropy, AKAZE keypoints, and Energy. Correlation analysis showed significant correlation between EEG features and image descriptors, validating that multimodal integration improves emotion recognition. The introduced deep-learning-based framework attained better classification accuracy than single-modality methods. The results underline the efficacy of the integration of neural, physiological, and visual cues and thus make the method worth considering for affective computing applications, mental health tracking, and human-computer interaction.