Development of Machine-Learning-Based Facial Thermal Image Analysis for Dynamic Emotion Sensing

Information on the relationship between facial thermal responses and emotional state is valuable for sensing emotion. Yet, previous research has typically relied on linear methods of analysis based on regions of interest (ROIs), which may overlook nonlinear pixel-wise information across the face. To address this limitation, we investigated the use of machine learning (ML) for pixel-level analysis of facial thermal images to estimate dynamic emotional arousal ratings. We collected facial thermal data from 20 participants who viewed five emotion-eliciting films and assessed their dynamic emotional self-reports. Our ML models, including random forest regression, support vector regression, ResNet-18, and ResNet-34, consistently demonstrated superior estimation performance compared to traditional simple or multiple linear regression models for the ROIs. To interpret the nonlinear relationships between facial temperature changes and arousal, saliency maps and integrated gradients were used for the ResNet-34 model. The results show nonlinear associations of arousal ratings in nose = tip, forehead, and cheek temperature changes. These findings imply that ML-based analysis of facial thermal images can estimate emotional arousal more effectively, pointing to potential applications of non-invasive emotion sensing for mental health, education, and human-computer interaction.