Emotion detection in artistic creation: A multi-sensor fusion approach leveraging biomechanical cues and enhanced CNN models

Artistic creation is a means of expressing human emotions. To intuitively capture the emotions conveyed by the artist in their works, we propose an improved CNN-based emotion detection method that incorporates biomechanical elements. Recognizing that emotions are accompanied by physiological and biomechanical responses such as heart rate variations, facial muscle activity, and speech tone fluctuations, we collect and integrate multi-sensor data, including heart rate, facial expression, and verbal expression. This information is processed through a multi-sensor signals fusion method based on an enhanced Convolutional Neural Networks (CNN), which allows for the extraction of rich and accurate emotional feature representations from the creator’s biomechanical signals. In particular, the facial muscle movements and subtle variations in speech tone, which are integral to understanding emotional states, are effectively captured and analyzed. Furthermore, we introduce a Conditioning Diffusion Model for Emotion Prediction, where emotional features, informed by biomechanical responses, serve as semantic conditions to boost the accuracy of emotion detection. This approach enables precise identification of the artist’s emotions by considering the intricate interplay of physiological and biomechanical signals. Experimental results demonstrate that our proposed method achieves an mAP score of 85.36%, an MSE score of 0.73%, and a runtime of 87 milliseconds, providing technical support for predicting the emotions of creators based on their biomechanical responses.