Adaptive multimodal transformer based on exchanging for multimodal sentiment analysis

Multimodal sentiment analysis significantly improves sentiment classification performance by integrating cross-modal emotional cues. However, existing methods still face challenges in key issues such as modal distribution differences, cross-modal interaction efficiency, and contextual correlation modeling. To address these issues, this paper proposes an Adaptive Multimodal Transformer based on Exchanging (AMTE) model, which employs an exchange fusion mechanism. When the local emotional features of one modality are insufficient, AMTE enhances them with the global features of another modality, thus bridging cross-modal semantic differences while retaining modality specificity, achieving efficient fusion. AMTE's multi-scale hierarchical fusion mechanism constructs an adaptive hyper-modal representation, effectively reducing the distribution differences between modalities. In the cross-modal exchange fusion stage, the language modality serves as the dominant modality, deeply fusing with the hyper-modal representation and combining contextual information for sentiment prediction. Experimental results show that AMTE achieves excellent performance, with binary sentiment classification accuracies of 89.18%, 88.28%, and 81.84% on the public datasets CMU-MOSI, CMU-MOSEI, and CH-SIMS, respectively.