Real-Time Emotion Detection in Live Video Streams using Multimodal Deep Learning Architectures

A new multimodal emotion detection system is introduced for use with live video feeds. The system integrates visual, auditory, and textual components and uses deep learning models for effective emotion recognition. EfficientNet, BERT, and LSTM with an attention-based fusion process. Unlike other single-modality methods that do not perform well in real environments, this framework self-adjusts the most significant mode to facilitate better performance in unpredictable situations. The system has low latency and real-time processing rates; hence, it can support applications including virtual meetings and e-learning platforms while live streaming. The performance of the introduced framework was tested on the MOSI dataset, and the model achieved an accuracy of 87.4%, an F1-Score of 86.2%, and an average latency of 45ms per sample. The latest study based on robustness testing shows a strong performance of 82.1% while operating in noisy conditions compared to other basic models, including Deep Affect Network, OpenSmile with Support Vector Machine, and the CMU Multimodal SDK. These results highlight the need to combine other modalities and utilize attention-based fusion to capture multifarious emotions. The problem of interest in this study addresses the three primary difficulties in emotion detection: multimodal fusion, noise tolerance, and real-time operation. It presents a new model for emotion recognition systems and the basis for future improvements, such as full model training and solving new types of emotional states. These results demonstrate the promise of this framework for furthering the science of emotion detection in complex, unstructured natural contexts.