Improving Multi-Fusion Strategies with Modality-Interactive Transformer for Emotion Recognition in Conversation

Recognizing emotions in conversations poses significant challenges due to the dynamic nature of emotions expressed across various modalities, including text, audio, and visual signals. It is critical to capture emotional cues and their context, yet many models ignore important information from audio and visual cues in favor of an excessive emphasis on text. As a result, emotional states are not fully understood. It is also more difficult to fully capture emotions while processing long conversations due to the large computational expenses involved. In order to address this issue, it requires more effective models that are capable of integrating emotional cues from every modality while maintaining optimal performance. A Modality-Interactive Transformer (MITr) is proposed to enhance the multimodal fusion strategies and representational ability of emotion recognition in multi-person conversations by giving equal priority to each modality. The MITr model captures both intra-modal and inter-modal interactions using a novel attention mechanism that combines sparse attention and relative positional encoding, reducing computational costs effectively. Additionally, an Emotional-State Module tracks speaker-specific emotional states while employing advanced feature extraction techniques to ensure accurate embeddings of unimodal data. Experiments on the IEMOCAP and MELD datasets validate the efficiency of MITr, outperforming state-of-the-art baselines in representational capability for multimodal emotion recognition.