BiMSA: Multimodal Sentiment Analysis Based on BiGRU and Bidirectional Interactive Attention

Currently, the significance of multimodal sentiment analysis is progressively increasing. However, the heterogeneity of multimodal signals poses a challenge in learning modality representation and fusing information. To address it, the BiMSA (multimodal sentiment analysis based on BiGRU and bidirectional interactive attention) is proposed in this paper. In the modality representation learning layer, BiMSA incorporates a structure consisting of BiGRU to extract contextual information from video and audio inputs. Subsequently, modality features are projected into modality internal representations and interactive representations for extracting information. This practice allows the model to take into account more aspects of modality information. In terms of the modality fusion layer, a bidirectional interactive attention mechanism is used to focus on the key representations of key modalities, and integrate multimodal information flexibly and efficiently. Attention weights are concentrated on modality representations that synergistically contribute toward overall sentiment orientation. Additionally, the constraints of similarity loss and difference loss are introduced to align with representations while mitigating redundant information and achieving a better fusion effect. Experimental results on public datasets (CMU-MOSI and CMU-MOSEI) demonstrate the effectiveness of the BiMSA model.