MSFR: Multi-Feature Skip-connection Fusion with Representation Learning for Multimodal Sentiment Analysis

In multimodal sentiment analysis tasks, we propose a novel method of multi-feature skip-connection fusion model based on representation learning (MSFR). The MSFR maps all feature representations extracted by raw videos into modality-invariant and modality-disparate subspaces using shared and private encoders. By this work, we learn modality-invariant and modality-disparate representations, which can narrow the heterogeneity gap among the three modalities. Additionally, emotional redundancy and conflicting information between modalities significantly impact the fusion effectiveness. To alleviate this problem, we design a text-centered multi-feature skip-connection fusion module. The final fused representations of modality-invariant and modality-disparate subspaces are used for the final sentiment analysis tasks. The proposed MSFR is extensively tested on two public benchmark datasets, CMU-MOSI and CMU-MOSEI. Compared to previous advanced methods, experimental results demonstrate the effectiveness of MSFR.