Optimizing Multimodal Emotion Recognition: Evaluating the Impact of Speech, Text, and Visual Modalities

This research aims to explore and optimize multimodal emotion recognition to enhance its performance. Multimodal emotion recognition involves analyzing information from different modalities—speech, vision, and text—to identify and classify emotional states accurately. This study investigates the roles of speech and text modalities and the auxiliary effect of visual modalities in multimodal emotion recognition. Study employs a two-stage multimodal information fusion neural network based on graph and attention mechanisms. The core idea is to achieve an effective fusion of multimodal information through graph convolutional networks and cross-modal attention mechanisms to improve emotion recognition performance. This study uses two datasets, Interactive Emotional Dyadic Motion Capture (IEMOCAP) and Multimodal Emotion Lines Dataset (MELD), to analyze the impact of speech and text modalities on visual modalities and their potential side effects. Results show that while the visual modality alone is less effective, adding speech or text significantly improves performance. However, introducing the speech modality and the text modality is less beneficial. Experimental outcomes reveal that the speech modality's effect is not as significant as the text modality, and its inclusion does not always lead to positive results, sometimes even degrading overall recognition performance.