Multimodal Transformer Fusion for Emotion Recognition using Facial and Voice Data

Nowadays, emotion recognition has become a crucial aspect of human-computer interaction for personalized and empathetic interactions. However, the existing Convolutional Neural Network (CNN) depended in a single modality which led to limited accuracy. Hence, this research proposes Multimodal Transformer Fusion (MTF) to recognize human emotions by combining the strengths of audio and image data. Initially, the input data is collected from Facial Emotion Recognition 2013 (FER2013) and Berlin Database of Emotional Speech (EmoDB) datasets which consists of facial images and speech signals. Then, the image data is preprocessed with histogram equalization and wiener filter to improve image quality and the signal data are preprocessed with Butterworth Bandpass Filter (BBF) respectively. After that, the Linear Binary Patterns (LBP) and Mel-Frequency Cepstral Coefficients (MFCC) are employed to extract the features from image and signal data respectively. Then, Convolutional Recurrent Neural Network (CRNN) and Bi-Directional Gated Recurrent Unit (Bi-Gru) are introduced to classify the emotions into seven types. Finally, the proposed MTF is employed to integrate the classified emotions and provides a final prediction of recognized emotions. From the results, the proposed MTF achieved better results when compared to CNN with Long Short-Term Memory (CNN-LSTM) in terms of accuracy (97.28%) respectively.