Deep Learning-Based Emotion Recognition using CNN, LSTM for Multimodal Text, Speech, and Facial Analysis

Emotion detection is a critical field in AI as well as computer vision, seeking to recognize or detect human emotions from facial expressions. Although this area has been greatly advanced through Convolutional Neural Networks (CNNs), current methods using only single-modal data often exhibit reduced accuracy. In this work, we confront these challenges with a hybrid deep learning method for disparate modalities of information such as speech, video, and text. We pre-process and then use CNN feature extractors for the FER2013 data set. The hybrid deep learning model, hence proposed, achieved a weighted accuracy of 79.3% on FER2013, overcoming the conventional unimodal methods. We are using transformers library with the pre-trained RoBERTa model for text. For speech emotion features, we are using TESS dataset then using a CNN and LSTM (Long Short-Term Memory) model. The CNN extracts feature from the speech, while the LSTM is used for temporal analysis to detect emotions. Once these multimodal features have been obtained, our work would be to fuse in order to get the final emotion recognition using Deep Neural Networks (DNNs). The process increases the precision and reliability of the recognition systems, helping to accumulate even more detailed intuition for different emotional states for a combined approach.