Emotionfusion: a Multimodal Approach to Emotion Detection Using Audio-Visual Cues

In this paper, we propose an upgraded model for emotion detection with the help of the RAVEDESS dataset which is multimodal and encompasses series of videos and audio files of emotion-eliciting scenes, including happiness, sadness, anger, fear, neutral, surprise and disgust. Regarding the multimodal data fusion, we utilize Convolutional Neural Networks (CNNs) since they are capable to handle spatial and temporal data. The audio files are then converted into spectrograms so that the CNN learns the important frequency based features that define the emotions. At the same time, the video sequences are extracted and transformed the frames with the help of CNN, which recognizes spatial features and vision at the micro level, including facial expressions and body language - in other words, emotions. This is done by training the CNN on the described multimodal inputs where the model implicitly identifies and correlates the patterns and details of audio and video to the particular emotional state. Finally, CNN based model is trained and fine-tuned through rigorous training for a higher degree of accuracy and robustness to the emotion expressed in the RAVEDESS dataset. The novelty of our proposed model upon the existing multimodal emotion detection is that we applied feature extraction and fusion processes to enhance accuracy and consistency of our model using CNN's models. The existing systems use simpler or less optimized fusion techniques which results in less accuracy of the model. The paper also discusses the advantages of CNNs in multimodal data analysis and proves the potential for further improvements in human-computer interaction, affective computing, and other areas where it is necessary to recognize and analyze human emotions.