Embedding Speech Temporal Information in Pretrained Language Models for Sentiment Analysis

Multimodal Sentiment Analysis is a task aimed at enabling computers to understand human emotions by learning representations from different modalities. Current mainstream methods primarily focus on cross-modal interactions, overlooking the impact of diverse data characteristics on sentiment analysis tasks. Therefore, our focus shifts towards the data itself, with the goal of extracting more nuanced sentiment representations directly from raw data. In this paper, a Dynamic Speech Temporal Module (D-STEM) is proposed, which employs three speech temporal enhancement steps to extract hierarchical temporal emotion representations from speech signals. Additionally, we dynamically embed the obtained speech temporal information into a pre-trained language model, resulting in the proposed speech-text sentiment analysis architecture. With just a few training steps, it can adapt to multimodal tasks. We evaluated our approach on the multimodal public sentiment datasets CMU-MOSI and CMU-MOSEI. Experimental results demonstrate that our method outperforms previous baseline models and text-only fine-tuning methods.