A survey of multimodal emotion recognition: fusion techniques, datasets, challenges and future directions

Emotion recognition is crucial in enhancing the quality of human-computer interaction, education, healthcare, and transportation safety. By integrating data from different sources, multimodal emotion recognition can capture complex emotional signals more comprehensively and accurately than single-modal data sources. This paper comprehensively reviews AI-based multimodal emotion recognition systems, covering deep learning techniques, datasets, challenges, and future research directions. We also present current research techniques, including exploring fusion strategies for different modal data and diverse data fusion methods. Several challenges in multimodal emotion recognition are discussed in this paper, such as the incompleteness of modal data, inconsistency in signal quality, and insufficient model interpretability. The paper points out that future research needs to further explore how to effectively integrate data from different modalities and enhance the adaptability and interpretability of models in practical applications.