Towards Athletes’ Emotion State Recognition with the Fusion of EEG and Facial Expression based on ResNet-LSTM

In sports science, understanding, regulating, and prioritizing athletes' emotional states are essential for enhancing performance and promoting well-being. This research attempts to overcome the weaknesses of unimodal methods of emotion recognition in adequately reflecting human emotion through the development of a new multiclass approach that integrates EEG signals using RGBD facial expression data. An experimental paradigm was designed to evoke three types of emotions- positive, neutral, and negative-to collect EEG signals from the 10 subjects in conjunction with a self-constructed RGBD facial expression dataset. EEG signals are processed via operations such as re-referencing and filtering, while the dataset is optimized with data alignment and denoising. A three-channel RES-LSTM network model was built using EEG signal data and the RGBD facial expression dataset, and further and efficiently combined dilated convolutions with attention mechanisms, with a significant improvement in emotion recognition being met. The optimized RES-LSTM model achieved a 93.12%- on average emotion recognition accuracy for 10 subjects, compared to single-modal emotion recognition, which showed remarkable promise on emotion recognition tasks.