Progressive Evolution of Emotion Detection: From Unimodal Baselines to a Quad-Modal Dynamic Fusion Architecture

BackgroundThe existing emotion detection systems are either audio or video centric.While such approaches have proven effective in controlled environments like a lab or a studio, these systems fail to perform under real-world conditions such as low light, audio interference, mispronunciation, and other environmental factors. ObjectiveThe objective of our research is to design a quad-modal emotion detection system that combines four sub-models (static facial images, dynamic facial expressions, speech tone, and natural language processing [NLP] of the spoken words) to design a robust model that can perform under real-world conditions. MethodsWe propose a dynamic weighting algorithm that biases the model towards a certain modality if the conditions are favorable (e.g.switch to audio-only if the video is obstructed) using cross-modal multi-head attention.We optimise the system to run on a CPU (due to the lack of a CUDA-enabled NVIDIA GPU) by using an asynchronous state buffer for the heavy NLP pipeline to maintain a high enough video frame rate. ResultsOur model achieved an accuracy of 61.21% (95% confidence interval [CI]: 58.77%-63.65%)with a mean area under the curve of 0.90 (95% CI: 0.88-0.92)across six emotion classes on real-world data (RAVDESS, SAMM, CREMA-D) during training and testing while maintaining performance under visual and audio deprivation conditions. ConclusionOur architecture, while forgoing the precision of a studio-lit, well-audio-recorded environment, is able to sacrifice some accuracy for the robustness required to function in a real-world human-computer interaction environment.