Multimodal Crisis Tweets Classification Enhanced by MLLMS and Dual-Channel Cross-Attention

The rapid growth of Twitter popularity has provided humanitarian organizations with novel information acquisition channels. Concurrently, advancements in multimodal information fusion techniques have enhanced the analytical accuracy of multimodal tweets. How to leverage the Multimodal Large Language Models (MLLMs) to further optimize the classification of multimodal crisis-related tweets represents a compelling research direction. Additionally, the challenge of refining multimodal fusion techniques to integrate supplementary knowledge with tweet information remains further explorations. Addressing these concerns, this paper presents the Commonsense-augmented Dual-channel Cross-Attention Network (CDCANet), a novel framework that integrates commonsense knowledge from MLLMs with feature fusion techniques. Our approach leverages MLLMs to generate image captions, which are combined with original tweet text and encoded using BERTweet to produce augmented textual representations. For visual feature extraction, we employ the ResNet to extract image features. The dual-channel cross-attention mechanism effectively fuses commonsense-augmented text features with image features. Experiments in the CrisisMMD dataset demonstrate the effectiveness of the proposed CDCANet.