A novel multimodal approach for emotion recognition using Text, Speech, and Facial Expression data

Multimodal emotion recognition (MER) is essential for improving human-computer interaction by allowing systems to comprehend and react to human emotions. This research introduces an innovative method for Multimodal Emotion Recognition (MER) by combining three modalities—text, speech, and facial expression—through a sophisticated framework that employs modality-specific feature selection, independent classification, and learnable attention-based feature fusion. The suggested approach tackles the difficulties of aligning and integrating features from several modalities while maintaining robustness and interpretability. The model adeptly adjusts to fluctuating emotional displays by utilizing the strengths of each modality. The research illustrates the efficacy of the suggested methodology via comprehensive trials, revealing substantial enhancements in classification precision and resilience relative to conventional unimodal and rudimentary multimodal systems. This research advances the creation of more empathic and contextually cognizant human-computer interaction systems, with potential applications in virtual assistants, mental health monitoring, and customer service.