A Multimodal Fusion Approach: Emotion Identification from Audio and Video Using Convolutional Neural Network

The ability to predict someone’s emotions goes beyond determining if they are happy or sad. It’s about peering into the future of a conversation. Humans express their emotions in a variety of ways, and understanding and interpreting them is not an easy task. While traditional methods focus on basic sentiment analysis, our work delves deeper, aiming to predict the nuanced emotional state of users during interaction. This paper proposes a multi-modal emotion recognition system with a simple late fusion model architecture that evaluates using speech emotion recognition and facial emotion expression methods. For both audio and video modalities, The architectures are designed with CNNs [1] and linear layers for prediction, achieving $\mathbf{9 7. 2 \%}$ accuracy on the RAVDESS dataset, classifying eight emotions. The outcome shows that these modalities provide useful information for identifying users’ emotional states.