Custom CNNs Based Multimodal Emotion Recognition from Facial and Speech Cues

This work introduces a comprehensive Multimodal Emotion Recognition (MMER) method that utilizes facial expression and speech data, using a custom 2-dimensional Convolutional Neural Network architecture for Face Emotion Recognition(FER) and, additionally, for Speech Emotion Recognition, custom 1-dimensional (1D) and 2-dimensional (2D) CNN models were crafted and tailored to analyze spectrograms and raw audio signals. The RAVDESS dataset, known for its diverse set of emotional expressions, was employed for training and evaluation. The proposed MMER system effectively integrates the information gleaned from both modalities, enhancing the accuracy and robustness of emotion recognition. Custom CNN architecture has shown promising results in combining facial and speech cues, highlighting the potential of multimodal approaches in discerning complex emotional states. This work contributes to the advancement of emotion recognition technology, with implications for applications in human-computer interaction, affective computing, and psychological research.