Cross-Modal Knowledge Distillation for Enhanced Unimodal Emotion Recognition

Multimodal physiological signal-based emotion recognition technology represents an emerging branch in the field of affective computing. By combining various physiological signals, its performance significantly surpasses traditional unimodal methods. Notably, the joint application of electroencephalography (EEG) and galvanic skin response (GSR) show high effectiveness in emotion recognition tasks. However, the difficulty and high cost of EEG signal acquisition limit its widespread use in practical applications. To address this challenge, we propose a Cross-Modal Emotion Knowledge Distillation (CMEKD) framework. This framework not only extracts the heterogeneity and interactivity between GSR and EEG signals but also conveys a comprehensive fusion of multimodal features into an unimodal GSR model through knowledge distillation techniques, thereby enhancing unimodal model performance. We employ cosine similarity-based knowledge representation to reduce the gap between the multimodal model and the unimodal model. Additionally, an adaptive feedback mechanism is introduced to dynamically adjust the distillation process based on the performance of the unimodal model, further improving the classification performance of emotion recognition. Experiment results demonstrate that this framework achieves state-of-the-art performance on two public datasets. By reducing reliance on multimodal data in the application of the emotion recognition technique, we significantly improve the practicality and feasibility of emotion recognition technology. The code for this paper will be made publicly available on GitHub: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://anonymous.4open.science/r/CMEKD-2440/</uri>.