SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition): The Emotion Whisperer—A Single-Stream Transformer for Multimodal Sentiment Analysis

Multimodal sentiment analysis enables machines to interact with people more naturally. The integration of multimodalities can enhance the machines’ ability to accurately predict emotions. The main obstacle to multimodal sentiment analysis is integrating information from different modalities. Previous research has used a variety of techniques, including long short-term memory networks (LSTM) and transformers. However, traditional fusion methods cannot better utilize the information from each modality, and some intra- and inter-modal features may be overlooked due to possible differences in feature representations. Therefore, to address this problem, we use a combined transformer that can connect different modal inputs and introduce SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition), a single-stream transformer that fuses textual, visual, and speech modalities. The model was pre-trained on the CMU-MOSI and CMU-MOSEI datasets: multi-modal masked image language modeling (MLM) and text–image matching (TIA). Compared to other existing models, SS-Trans improves ACC-2 on these two datasets by 1.06% and 1.33%, and improves F1 values by 1.50% and 1.62%, respectively. The experimental results show that our method achieves the state-of-the-art level. In addition, ablation experiments validate the model and the pre-training task, proving the effectiveness of the proposed model.