Bridging human emotion processing and deep neural networks: insights from representational similarity analysis

Emotion is a complex psychophysiological response to external stimuli, essential for human survival, social interaction, and human-computer interaction. Emotion recognition plays a critical role in both biological systems and artificial agents. However, existing research often treats these systems independently, limiting opportunities for interaction and hindering the development of more advanced models. This study employs representational similarity analysis (RSA) to bridge this gap by comparing emotional representations between the human brain and neural networks, aiming to improve understanding of emotion recognition in deep learning models. By correlating the emotion recognition model EmoNet with EEG signals from the human brain during emotional image processing and introducing AlexNet for comparison, we reveal EmoNet’s human-like representation for emotional images and its hierarchical structure for emotion recognition. The results show that RSA effectively aligns human emotional processing with deep neural networks, offering new avenues for improving the interpretability and performance of emotional AI models. Moreover, they underscore EmoNet’s potential to simulate human emotional responses, paving the way for future research to enhance emotion recognition models by incorporating human emotional evaluations into their training processes, thereby improving efficiency and specificity.