Development of emotion recognition for rehabilitation feedback system using wavelet transform and LSTM

Abstract Human assist devices have been used to assist users in rehabilitation environments to correct patients’ motions to increase the rehabilitation quality. However, they are usually a routing working machine that cannot consider the patient’s emotions. By introducing emotion recognition into the system, the device can detect the user’s condition more accurately and provide a corresponding action to improve the experience. This study focuses on enhancing the precision of emotion recognition for a rehabilitation device designed for stroke patients, which will adjust according to their emotional state to improve their rehabilitation experience. In this study, the Database for Emotion Analysis using Physiological Signals (DEAP) is leveraged, which includes multimodal physiological signals and subjective emotion ratings from 32 participants watching 40 emotion-evoking video clips that is suitable for neural network training. Two preprocessing methods, Short-Time Fourier Transform (STFT) and Wavelet Transform (WT), are considered for utilizing the dataset. Additionally, DNN, DenseNet, and Long Short-Term Memory (LSTM) networks are considered the core of the classification method, which will determine the emotion into 3, 5, or 9 categories. The result indicates that STFT and WT preserve temporal and frequency information that can benefit the classification. Among the models, LSTM demonstrates superior capability in capturing sequential data features from EEG signals, leading to higher accuracy rates. Throughout experiments, the best performance was achieved with the LSTM network processed with WT, with accuracies of 88% for valence and 90% for arousal in $$3\times 3$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>3</mml:mn> <mml:mo>×</mml:mo> <mml:mn>3</mml:mn> </mml:mrow> </mml:math> classification, which is better than the previous system. This comprehensive analysis underscores the importance of choosing appropriate signal processing techniques and neural network architectures to enhance the precision and reliability of emotion recognition systems using EEG data. It can be further applied to detect patients’ conditions to provide a better rehabilitation experience for stroke patients.