Manifold knowledge-guided feature fusion network for multimodal sentiment analysis

With the continuous progress of multimedia and information technology, multimodal sentiment analysis (MSA) has become one of the most advanced and challenging research directions in the field of artificial intelligence . Multimodal data, including text, visual and audio information, provides additional perspectives for sentiment analysis. However, extraneous information in non-verbal modalities affects the accuracy of sentiment analysis, as sentiment-related features are mainly concentrated in changes in mouth movements and pitch changes, which poses a challenge for accurate sentiment analysis. To solve this problem, we propose a manifold knowledge-guided feature fusion network (MKGN). MKGN uses manifold knowledge generated by manifold learning algorithms to guide neural networks to extract effective non-verbal features and establish associations between multiple features while reducing dimensionality. In addition, in order to improve the quality of knowledge, we propose two knowledge enhancement methods: knowledge filter (KF) and knowledge contrastive learning (CL). Among them, KF is used to filter out unreliable knowledge, and CL further strengthens retained knowledge by changing the distance between knowledge. Importantly, the proposed MKGN achieves excellent performance on three datasets compared to state-of-the-art models. On the MOSI dataset, the accuracy is improved by 2% and 1%, respectively. On the MOSEI dataset, the accuracy improved by 3.8% and 1.8%, respectively. On the UR-FUNNY dataset, the accuracy improved by 0.4%. • Propose a feature fusion network based on manifold learning. • The generated manifold knowledge reduces redundant features. • The contrastive learning enhances the manifold knowledge. • The proposed model achieves excellent results in three datasets.