Facial Dynamic Based Multimodal Face Recognition and Emotion Analysis Using Multiscale Residual Dense Attention Network

Recognition of the human emotional state is a complex psychological phenomenon, which is difficult to determine accurately using a single modality. Most of the previous research on the multi-face expression detection approach utilizes the multi-modality, in which the standard model leads to insufficient generalization and poor real-time performance. Additionally, the earlier facial expression dataset includes lowquality facial images, which lack specificity and also limited the development of the investigation. The novelty of this article is to design a multimodal face recognition and emotion analysis framework using the latest deep learning schemes. Initially, the various multimodal inputs like face images, audio, and Electroencephalography (EEG) signals are processed by the proposed model to achieve effective emotion recognition. From each modal input, three separate sets of features are extracted. The features from the face image are constituted as feature set 1, and the Mel Frequency Cepstral Coefficient (MFCC) obtained from audio signals and Short-Time Fourier Transform (STFT) features from EEG signal are constituted as feature set 2 and feature set 3. The entire set of features is provided to the Multiscale Residual Dense Attention network (MRDANet) for training, and identifying different classes of emotions from the data. Evaluation of the proposed scheme is done under different setups and is compared to different state-of-the-art techniques.