MSA-GCN: A Graph Convolutional Network Approach for Internal Feature Fusion in Multimodal Sentiment Analysis

Multimodal Sentiment Analysis (MSA) improves the accuracy of sentiment prediction by fusing three modalities: text, audio, and visual. However, we found that there are two key limitation problems prevalent in the existing methods: on the one hand, the existing techniques overly focus on the realization of inter-modal interactions but neglect the mining of spatiotemporal features within individual modalities; on the other hand, the traditional fusion strategies are not refined enough to effectively model fine-grained spatiotemporal dependency pathways. Therefore, to solve the above two problems, we propose a multimodal sentiment analysis framework based on graph convolution network (GCN), MSA-GCN(Multimodal Spatiotemporal Attention Graph Convolutional Network), which strengthens the modal internal feature representation through the hierarchical graph convolution mechanism, and innovatively constructs an independent spatiotemporal graph model, so that cross-modal fusion can be realized while fully exploiting the spatial and temporal features of single modality. The experimental results show that MSA-GCN significantly outperforms other baseline algorithms in several metrics, especially in the long-time sequence emotion recognition task.