Multimodal Emotion-Cause Pair Extraction with Holistic Interaction and Label Constraint

The multimodal emotion-cause pair extraction (MECPE) task aims to detect the emotions, causes, and emotion-cause pairs from multimodal conversations. Existing methods for this task typically concatenate representations of each utterance from distinct modalities and then predict emotion-cause pairs directly. This approach struggles to effectively integrate multimodal features and capture the subtleties of emotion transitions, which are crucial for accurately identifying causes—thereby limiting overall performance. To address these challenges, we propose a novel model that captures holistic interaction and label constraint (HiLo) features for the MECPE task. HiLo facilitates cross-modality and cross-utterance feature interaction with various attention mechanisms, establishing a robust foundation for precise cause extraction. Notably, our model innovatively leverages emotion transition features as pivotal cues to enhance causal inference within conversations. The experimental results demonstrate the superior performance of HiLo, evidenced by an increase of more than 2% in the F1 score compared to existing benchmarks. Further analysis reveals that our approach adeptly utilizes multimodal and dialogue features, making a significant contribution to the field of emotion-cause analysis. Our code is publicly available at https://is.gd/MVdYmx .