pSCOUTER: Time-Series Emotion Classification Using Contactless Measured Multimodal Biosignals in Medical Diagnosis

Effective communication between doctors and patients significantly impacts the quality of diagnosis and treatment, especially in emotionally challenging situations such as cancer diagnosis. This study aims to enhance emotion recognition in clinical settings by providing doctors with feedback on patients' emotions using non-contact multimodal biosignals and an LSTM model. The pSCOUTER system was used to non-invasively collect heart rate and respiratory signals from patients during simulated medical consultations for three scenarios: initial cancer diagnosis, recurrence/metastasis diagnosis, and terminal cancer diagnosis. The collected signals were segmented into 5-second intervals, from which 20 features were extracted. These features were used to train both the LSTM and GPT4-omni models. Emotion labels were classified into seven categories based on Russell's circumplex model of affect: Happy, Relaxed, Disgust, Sad, Upset, Fear, and Surprised.As a result, the LSTM model outperformed both the GPT4-omni model and doctors' predictions in terms of F1 score, demonstrating the highest accuracy across all scenarios. This suggests that LSTM networks are highly effective in capturing the temporal dependencies in multimodal physiological data. This study highlights the potential of integrating non-contact biosignal measurement with machine learning techniques to improve emotion understanding in medical settings, ultimately aiming to enhance patient care and satisfaction.