Multimodal Analysis for Speech Emotion Recognition

Speech Emotion Recognition (SER) is becoming an essential field of study in today’s time due to its applications in customer service, enhancing human-computer interaction, and healthcare. However, SER is challenging due to variations in emotional expression across individuals, overlapping tones for different emotions, and limited datasets. The study focuses on recognizing emotions such as anger, happiness, sadness, fear, surprise, disgust, calmness, and neutral. We combine four popular datasets: CREMA-D (Crowdsourced Emotional Multimodal Actors Dataset), featuring speech from 91 actors; RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song), with recordings from 24 actors for multimodal approaches; SAVEE (Southern Alberta Vocal Expression Emotion), containing emotional speech from 4 male actors; and TESS (Toronto Emotional Speech Set), including speech samples from 2 female actors. The study implements three approaches: Convolutional Neural Networks (CNNs) for feature extraction, Long Short- Term Memory (LSTM) networks for recognizing temporal patterns, and Bidirectional LSTMs (Bi-LSTMs) with Multi head Attention for enhanced contextual understanding. Our proposed system achieves improved accuracy in SER, setting a foundation for its application in domains such as healthcare, customer service, and human-computer interaction.