HGTFM: Hierarchical Gating-Driven Transformer Fusion Model for Robust Multimodal Sentiment Analysis

In multimodal sentiment analysis, a significant challenge lies in quantifying the contribution of each modality and achieving effective modality fusion. This paper presents a Hierarchical Gating-Driven Transformer Fusion Model (HGTFM), which effectively achieves multimodal data fusion through an advanced transformer architecture and a Hierarchical Gated Fusion Module (HGFM). The model first encodes audio and video features using LSTM and CNN, respectively, while employing BERT to encode text features. This approach effectively enhances the model ability to manage long-range temporal dependencies. Furthermore, the model introduces HGFM, which dynamically integrates multimodal features across different levels after processing by the modality interaction network (MIN), thus intelligently adjusting the contribution weights of each modality. In particular, the model incorporates a Unimodal Label Generation Module (ULGM), which considers the differences between the modalities. This enables HGFM to more accurately identify the contributions of samples with significant modal variations, thus optimizing the module potential. The experimental results on several established benchmark datasets demonstrate that HGTFM achieves comparable performance improvements in multimodal sentiment analysis tasks.