Multi-Modal Patients Emotion Recognition Framework Using Dual Attention Based Bi-GRU and Ensemble Features in the Healthcare Industry

Accurate recognition of patient emotions is helpful for the identification of the emotional state of patients like anger, stress, and sadness, which is useful to provide patient communication. Hence, better clinical training is provided based on the classified emotions. In addition, videos taken from a particular distance make several misclassifications in outcome while recognizing emotions. Therefore, a novel emotion recognition model based on the integration of multiple modalities is crucial for recognizing the patient's emotions. Hence, an efficient patient emotion recognition technique is designed by considering deep learning techniques. In the initial stages, multimodal data like text, video, and Electroencephalogram (EEG) signals are collected from benchmark resources. Next, the multimodal information is offered in the feature extraction phase. Here, the significant features presented in the text are extracted through the Term Frequency-Inverse Document Frequency (TFIDF), and the Video is taken by the 3-Dimensional Visual Geometry Group (3D-VGG16). From EGG signals, Spectral Features and Wave features are acquired. Next, the extracted three different features are offered to the novelty model Dual Attention based Bidirectional-Gated Recurrent Unit (DA-Bi-GRU)-aided patient's emotion recognition phase. The usage of the dual attention mechanism in Bi-GRU only focuses on more relevant information from the ensemble features and the complex relationship between these features is attained thus improving the recognition performance of emotional states. The ensemble features-based emotion recognition effectively classifies the different states. Finally, the patient's emotion-recognized outcomes will be obtained from developed DA-Bi-GRU. Later, different performance analysis is executed in the developed framework over the classical techniques to verify its efficiency.