A Brain-Inspired Multimodal Sentiment Analysis Framework via Rationale-Guided Representation

Multimodal sentiment analysis (MSA) recognizes human sentiments with various data modalities. Existing works primarily focus on efficient feature extractors for each modality or multimodal fusion frameworks. However, they do not exploit the sentiment-related prior knowledge in the human brain, limited in their performance when sentiment cues tend to be more implicit and ambiguous. To address this problem, we propose a rationale-guided prompt learning optimization framework (RaPo) inspired by the sentiment chains of biological brains. Specifically, we adopt a chain-of-thought prompt to analyze images with a large visual-language model, generate the corresponding contextual captions and rationales, which are then combined to derive the sentiment prompt. Finally, the prompt is used to optimize the RaPo framework through the designed rationale-guided prompt tuning. Experiments on several MSA tasks consistently show an outperformance of several state-of-the-art methods, with an average increase in accuracy by 2.8%.