Multimodal emotion recognition in conversation based on speaker dependency

This paper, addressing the lack of interaction between context and modalities and the deficiency in speaker dependency in dialogue sentiment analysis, proposes a multimodal fusion method based on cross-modal attention. This method leverages cross-modal attention to facilitate pairwise interaction of textual, audio, and visual modality information. Additionally, it introduces a Speaker Dependency Encoder (SDE) network that divides historical dialogues and speaker features according to the identity of the speakers. This network divides historical conversations and speaker features according to speaker identity, and uses attention mechanism to explore inter-speaker and intra-speaker dependence from the divided conversations and speaker features respectively. The above dependencies are then spliced to obtain the speaker dependencies at the current moment. A multimodal dialogue sentiment analysis network, Speaker Guide Emotion Network (SGEN), is proposed, which combines the modal fusion network CMA and speaker feature generation network SDE proposed in this paper to undertake the task of multimodal dialogue sentiment analysis.