Feature Importance and Model Performance in Deep Learning for Speech Emotion Recognition

Speech emotion recognition (SER) is detecting a person’s emotional state through their voice, a technique now widely applied across various fields. This paper aims to explore the significance of different acoustic features in SER using the TESS dataset, which comprises approximately 2800 audio files featuring 200 words spoken by two female actors, representing seven emotions: anger, disgust, fear, happiness, pleasant surprise, sadness, and neutrality. We extracted seven key acoustic features from these audio files: Chromas, Mel Frequency Cepstral Coefficients (MFCCs), Mel spectrogram, Zero crossing rate, RMS power, contrast, and pitch. Our study involved applying LSTM and CNN deep learning methods to assess the performance of each feature. Results indicate that the MFCC feature performs better, with accuracy exceeding 99% in LSTM and CNN models. Additionally, we employed traditional machine learning models, specifically random forest and decision tree classifiers, combined with a grid search feature selection method to generate feature importance graphs. These results highlight pitch as the most critical feature in both models. However, pitch alone yielded only 28% accuracy in LSTM and CNN models. This discrepancy underscores the complex interplay between feature selection and model architecture in optimizing SER performance. Our findings reveal critical insights into the interplay between feature relevance and model efficacy, highlighting the necessity of feature optimization for enhancing SER systems.