MMER: Mid-Level Fusion Strategy for Multimodal Emotion Recognition using Speech and Video Data

Multimodal emotion recognition presents a significant challenge still it holds immense potential for improving human-computer interaction across diverse applications. This study aims to develop an efficient framework for classifying emotions using multimodal data containing visuals and audios. For this purpose, a novel approach called the Mid-level Fusion Strategy for Multimodal Emotion Recognition (MMER) is utilized which incorporates deep learning based feature extractor networks for the audio and video data proceeded with the mid-level fusion strategy. The process involves custom four-layer CNN architecture designed for managing both audio and video data, utilizing Conv2D layers for audio data and Conv2DLSTM layers for video data. The results and experimentation revealed that the MMER framework achieved high performance on the RAVDESS and SAVEE datasets, with accuracy of 86% and 91% respectively, indicating significant improvements over the existing methods in multimodal emotion recognition.