Two-Stage Multi-Modal Prompt Tuning for Few-Shot Sentiment Analysis

Few-shot multi-modal sentiment analysis (MSA) is a crucial task in the field of visual-language understanding and playing a pivotal role across various application domains (e.g., interaction, e-commerce promotion and social media analysis, etc). Recently, with the development of pre-trained language models, prior works have predominantly utilized a combination of pre-trained language models alongside visual encoders and employed prompt learning to generalize pre-trained language models to the MSA tasks. However, there are specialized vision-language pre-trained models (VLPM) designed to handle visual-language tasks, such as visual question answering. There are little explorations on VLPM and its prompt learning method for multi-modal sentiment analysis. Therefore, our work fills this gap by proposing the Two-stage Multi-modal Prompt Tuning (TSMMP) for few-shot sentiment analysis based on VLPM. TSMMP consists of two stage prompt tunings. In the first stage, we encode the images and text separately and feed them into a VLPM to obtain three outputs: Multi-modal Prompt (MM), Sentiment Answer (ANS), and Visual Prompt (VP). Subsequently, we design a parameterized fusion module to merge the three outputs. The merged representation is served as initialization for the prompts in the second stage. We design a multi-modal prompt approach for the second stage, which consists of Manual Prompts, Learned Prompts, and Fusion Prompts. Manual Prompts are manually crafted natural language templates, Learned Prompts are a series of continuously trainable prompt vectors and Fusion Prompts are the result of merging MM, ANS, and VP via the fusion model. Through thorough experimental analysis, we demonstrated that our method performs well on both coarse-grained and fine-grained datasets. By conducting ablation studies, we have shown that each component of our proposed multi-modal prompt template is effective, and the two-stage approach outperforms the one-stage approach entirely.