SKE-MSA: Enhancing Representation Learning with VAD Lexicon for Multimodal Sentiment Analysis

Most existing Multimodal Sentiment Analysis (MSA) models that are enhanced with external sentiment knowledge primarily focus on integrating this knowledge during the multimodal fusion stage, while overlooking its potential benefits during the multimodal representation learning process. In this study, we propose a Sentiment Knowledge-Enhanced Multimodal Sentiment Analysis (SKE-MSA) framework, which incorporates an external sentiment knowledge base—the VAD lexicon—to enhance MSA. SKE-MSA introduces the Sentiment-aware Encoding Loss (SAE_Loss), designed to leverage external sentiment knowledge to guide the representation learning of multimodal encoders. Building on this, we extract both common and unique features of multimodal representations and subsequently predict sentiment intensity based on these features. Extensive experiments conducted on three MSA datasets demonstrate the competitive performance of SKE-MSA.