Emotion Recognition from Audio, Live Video and Text

Specific user experiences, behavioral analysis and human-computer interaction all depend extensively on emotion recognition. However, because patterns of speech, gestures and linguistic context vary it is still difficult to accurately recognize emotions in text, audio and video. Because real-time inputs are dynamic, noisy and have small datasets traditional methods frequently have trouble applying. This paper suggests a strong mixed emotion identification system that combines deep learning methods for text-based emotion detection, audio and live video in order to overcome these difficulties. Four different Kaggle datasets were used for the audio and complex audio augmentation methods like speed modifications, stretching and shifting were used to improve model stability. Using extracted features a Convolutional Neural Network (CNN) was trained and it showed amazing accuracy of 99.44 % and validation accuracy of 94.71 %. The CNN-based model was able to successfully learn facial expressions in live video to identify emotions by using OpenCV for real-time image processing and picture data augmentation. The model demonstrated the difficulty of real-time variance by obtaining a high accuracy in training of 97.66 % but a validation accuracy of 65.38 %. Six emotions joy, sorrow, anger, fear, love and surprise were rated using a BERT model for text-based emotion recognition and features were extracted using TF-IDF. Flask was used to deploy the system for real-time emotion recognition providing a smooth and effective user experience. By combining these methods emotion detection precision and dependability are improved providing a complete solution for applications in intelligent recommendation systems mental health evaluation and interaction between humans and computers.