Multimodal Facial Expression Recognition using Long Short-Term Memory with Position Attention Mechanism

Multimodal Facial Expression Recognition (FER) integrates multiple modalities such as audio, video and text data to enhance the accuracy through capturing various aspects of human expressions. However, accurate recognition of facial expressions among audio, video and text modalities remains challenging because of a variation in expression representation. Therefore, this research proposes a Long Short-Term Memory with Position Attention Mechanism (LSTM-PAM) for multimodal FER. Initially, the Surrey Audio-Visual Expressed Emotion (SAVEE) and Multimodal Corpus of Sentiment Intensity (CMU-MOSI) datasets are used which has audio, video and text modalities. Then, the wiener filtering is applied for audio data, Contrast-Limited Adaptive Histogram Equalization (CLAHE) is used for video data and lemmatization for text data which enhances the FER performance through reducing noise, improving contrast and standardizing textual data. The Mel-Frequency Cepstral Coefficients (MFCC) is applied for audio data, Gray-Level Co-Occurrence Matrix (GLCM) is used for video data and Word2Vector for text data which captures spectral, texture and semantic features for better accuracy in FER. Then, these extracted multimodality features are combined into a single vector. At last, the LSTM-PAM is applied for FER which provides the network to allocate various spatial positions in face images thereby enhancing the accurate recognition. The LSTM-PAM achieved accuracy of 98.25% and 95.78% for SAVEE and CMU-MOSI datasets which is better than existing approaches.