Emotion Recognition from RGB Kinect Videos and Physiological Signals Using Pre-Trained CNN Model

This study presents an innovative approach to emotion recognition by integrating RGB Kinect video data with physiological signals, including electroencephalography (EEG), electrocardiography (ECG), and galvanic skin response (GSR). Using the AMIGOS dataset, which includes video recordings and self-assessment emotional ratings, we improve emotion prediction through a comprehensive multimodal analysis. Our method involves feature extraction and averaging from both video and physiological signals, emphasizing the connection between visual cues and emotional responses. A pre-trained ResNet model is used for advanced feature extraction from video data. Various machine learning algorithms are applied to evaluate predictive models against self-assessment labels. Results indicate that the combination of ResNet-based visual features and physiological signals enhances emotion prediction accuracy, underscoring the value of a multimodal approach in capturing complex emotional dynamics.