Multimodal Emotion Recognition based on Convolutional Neural Networks and Long Short-Term Memory Networks

Due to the rapid advancements in big data and high-performance computing technologies, the application of deep learning models in the fields of computer vision and signal image processing has become increasingly widespread. To address the low efficiency and accuracy in the recognition of multimodal emotional EEG signals, this study integrates the latest deep learning network models, leveraging the advantages of convolutional neural networks (CNNs) in EEG feature extraction and the strengths of long short-term memory (LSTM) networks in processing dynamic temporal information. Therefore, this study employs CNN models to extract features from multimodal EEG signals, followed by feature vector fusion using LSTM, and classification using softmax. The network model constructed in this study achieved an accuracy of 92% in the three-class emotion recognition. Compared with traditional classification methods, the CNN-LSTM network model exhibits higher recognition efficiency due to its combined advantages in feature extraction and dynamic temporal information processing. Compared to using CNN or LSTM models alone, this model also shows a significant improvement in recognition accuracy.