FP-KDNet: Facial Perception and Knowledge Distillation Network for Emotion Recogniton in Coversation

Emotion recognition in conversation (ERC) is anchored in the burgeoning field of artificial intelligence, aiming to equip machines with the ability to discern and respond to human emotions in nuanced ways. However, recent studies have primarily focused on textual modalities, often neglecting the significant potential of non-verbal cues found in audio and video, which are critical for accurately capturing emotions. Furthermore, when researchers integrate these non-verbal cues into multimodal emotion recognition systems, they encounter challenges related to data heterogeneity across different modalities. This paper introduces the Facial Perception and Knowledge Distillation Network (FP-KDNet) to address these challenges. Specifically, a novel Facial Perceptual Attention (FPA) module was designed to capture non-verbal cues from videos, significantly enhancing the model’s ability to process visual information. Additionally, a knowledge distillation (KD) strategy was proposed to improve emotion representation in the non-verbal modality by leveraging data from the text modality, facilitating effective cross-modal information exchange. A multi-head attention mechanism further optimizes the integration of features across modalities, dynamically adjusting attention allocation to enhance conversational emotion recognition. The experimental results demonstrated that FP-KDNet achieves excellent performance on the MELD and IEMOCAP datasets, and ablation studies confirm the effectiveness of the multimodal fusion approach.