Personal Behavior-Robust and Interpretable Multimodal Sentiment Analysis

Sentiment analysis, crucial in fields like human computer interaction and mental health, has advanced through multimodal fusion and large-scale language models. People exhibit significant variability in emotional expressions due to behavioral differences, yet existing methods often apply generic models across populations. By focusing primarily on objective features, these models overlook “personal” cues that may lead to inconsistencies between the extracted features and the corresponding emotion categories. Additionally, at the common level, this issue is aggravated by the accumulation of personal cues mapped to a common emotion category. Existing approaches tend to fuse these cues indiscriminately, whereas features that align consistently with the target emotion category should be emphasized. And the attention mechanism, as the dominant fusion method, cannot adjust the misallocated weights it assigns. To address these limitations, we propose a “personal behavior-robust” sentiment analysis approach that disentangles personal cues from common ones through decoupling, and learns the relative importance of features across modalities via a corrective feedback-based “interpretable” fusion strategy to adjust misallocated weights assigned by the attention mechanism. The proposed model effectively differentiates and prioritizes relevant information from multiple cues, guiding the model to focus on the cue that aligns with the overall emotion. Experimental results demonstrate that the proposed methods achieve good performance on two public datasets: the CMU-MOSI, and CMU-MOSEI datasets.