Research on a multimodal emotion perception model based on GCN+GIN hybrid model

Graph neural networks (GNNs) have demonstrated strong performance in handling graph-structured data in recent years, particularly in capturing complex inter-node relationships among data samples, showcasing advantages over traditional neural networks. However, challenges persist, including difficulties in cross-modal information fusion, inadequate modeling of modal relationships, and high computational costs. To address these limitations, this paper proposes GGMEN, a novel model that integrates the local neighborhood aggregation capability of graph convolutional networks with the global structural expressiveness of graph isomorphic networks (GINs). Leveraging shallow feature extraction via time-frequency joint analysis, the paper extracts 14 representative physiological statistical features. Simultaneously, the Transformer model captures spatial features from individual facial expression video frames, enabling spatio-temporal modeling of facial expressions. The graph convolutional network (GCN) layer models temporal dependencies in physiological signals and spatial relationships of facial key points, while the GIN layer enhances modeling of complex higher-order relationships. Multi-modal emotion perception is achieved through attention-based modality fusion. Experiments on multiple datasets validate the model’s effectiveness across multiple emotion perception benchmarks, achieving an emotion recognition accuracy of 81.25% on dataset DEAP. Comparative analyses with existing models confirm the accuracy improvement of the proposed framework.