Ensemble Features for Multi-Modal Data Based Patient Emotion Recognition Scheme Using Multiscale LSTM with Sparse Attention Network

Presently, emotion recognition techniques are used in different domains like rehabilitation and neuroscience, due to their better performances and efficiency. The classical emotion recognition techniques didn't offer more focus on analyzing the emotions among depressed patients who had undergone medical treatments. Here, a novel framework is suggested to recognize the emotions of the patients using multi-modal data. The novelty of the developed framework is to detect the mental health conditions of individuals using multiple features and deep learning techniques. Here, the recommended M-LSTM-SA enhances the overall robustness and accuracy of emotion recognition. In the beginning phase, text is given as the input to the Term Frequency-Inverse Document Frequency (TF-IDF) to extract the first set of features. Next, video sequences are provided to the Residual Network (ResNet) to acquire the second set of features. Next, the Electroencephalogram (EEG) signals are converted into spectrogram images, and then the third set of information is collected using the ResNet technique. Next, the acquired ensemble features are given to the Multiscale Long Short-Term Memory with Sparse Attention (M-LSTM-SA)-based multimodal patient's emotions recognition phase. The developed M-LSTMSA offered the multimodal patient's emotions recognized outcomes and then, various performance measures were considered to observe the performance of the developed scheme over classical frameworks.