Enhancing Emotional Understanding through Multimodal Fusion: A Dual Channel Approach with Self-Attention Mechanism and Transfer Learning

The understanding of emotions plays a pivotal role in a multitude of fields, including human-computer interaction, affective computing and human behaviour analysis. The intricacy and subtlety of emotional states are often challenging to capture with traditional single-mode approaches. With the advent of eye tracking technology and facial expression recognition, eye tracking and facial expression can be employed in conjunction with one another, collectively offering insights into emotions. The integration of these two types of information can provide a more comprehensive and accurate description of an individual’s emotional experience. A data collection experiment was designed using emotion theory in psychology. Three types of emotionally activated images (positive, neutral, and negative) were selected from the Chinese Emotion Image System (CAPS). In order to better capture the correlation and importance between different modal information, a system was designed to dynamically adjust the weights of different modal features through feature fusion using Self-Attention mechanism. A two-channel multimodal emotion recognition model based on Residual Neural Network (ResNet) and Convolutional Neural Network (CNN) is proposed. The network was initialised and trained on a large dataset using a transfer learning approach, and then tested by fine-tuning on the collected data. Finally, the model achieved an accuracy of ${9 6 . 2 8} \%$. The combination of eye movement signals and facial features improves the accuracy and robustness of emotion recognition compared to modality-based emotion recognition methods.