Multimodal Emotion Cause Pair Extraction in Conversations Using Knowledge Distillation and Large Language Models

This paper presents a novel approach to Multimodal Emotion Cause Pair Extraction in Conversations (ECPEC), by addressing the complexities of emotion recognition by integrating features from different data sources. The method consists of a two-stage process. In the first context modeling stage, context-rich summaries of conversations filled with emotion cues are generated using a teacher-student knowledge distillation approach, lever-aging a Large Language Model (LLM) as the teacher and fine-tuning smaller LLM to mimic its summarizing capability. The generated summaries serve as input for the second stage, where the linguistic, acoustic, and visual features are fused to extract both emotions and emotion-cause pairs. To enhance the learning process, Direct Preference Optimization (DPO) is employed to fine-tune the compact LLM model by creating a preference dataset using a Ground Truth Augmented Reasoning technique. The responses from the fine-tuned LLM are converted into structured outputs using Mistral 7B model facilitating objective evaluation through the traditional performance metrics used in classification tasks. The findings demonstrate the critical role of multimodal data in capturing emotions, by enhancing the role of each modality through the reasoning capabilities of the LLM models. This work establishes a robust framework for ECPEC, demonstrating the effectiveness of integrating multimodal cues and advanced LLM fine-tuning strategies in affect computing tasks.