A Multimodal Approach for Persian Speech Emotion Recognition with Evolutionary Feature Selection

Understanding emotions in speech is fundamental to natural human-computer interaction, yet accurately capturing these emotions remains a challenge, especially in low-resource languages like Persian. Linguistic complexity and limited labeled datasets hinder the performance of conventional Speech Emotion Recognition (SER) models. To address these challenges, this study introduces a multimodal SER system that integrates acoustic and textual features for improved emotion classification. The model employs Whisper ASR (Automatic Speech Recognition) for speech-to-text conversion and a Modified Differential Evolution (MDE) algorithm for optimized speech feature selection. Extracted features are fused using a self-attention mechanism before being processed by a deep learning-based classification model. Evaluations on the ShEMO (Sharif Speech Emotion) dataset demonstrate that the proposed approach achieves significant improvements over existing methods. Results highlight the impact of multimodal integration and evolutionary optimization in refining SER performance, particularly in handling complex emotional expressions. This research advances emotion recognition technology for applications in virtual assistants, mental health monitoring, and customer service automation.