MSAF: Multimodal Sentiment Detection via Multiscale Adaptive Fusion

With the rapid increase of multimodal comments on social media, multimodal sentiment detection has become increasingly important. However, most existing methods overlook the difference in information density between text and images, and fall short in fully utilizing multi-scale information in images. To address this issue, we propose a multi-scale adaptive fusion model termed MSAF for multimodal sentiment detection. MSAF first extracts fine-scale and coarse-scale features of images through a multi-scale visual encoder, and uses a multi-scale adaptive pooling module to adaptively adjust the weights of different regional features. Then, MSAF incorporates multi-scale contrastive learning and multi-scale rivalry tasks to ensure that the model retains associations between features at different scales while maintaining their diversity. These features are sequentially fused with text through a hierarchical fusion encoder guided by textual information, enabling MSAF to focus on sentiment-salient regions in the image. Finally, the multimodal fusion embeddings are fed into a classifier to predict the sentiment. Extensive experiments on multiple public datasets demonstrate the effectiveness and superiority of MSAF. The codes of MSAF are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ADMIS-TONGJI/MSAF</uri>.