Advancing speech emotion recognition with whisper model embeddings and hand-crafted audio descriptors

The variability in speech patterns and limitations of conventional feature extraction methods continue to pose challenges for speech emotion recognition (SER). Large language models have become available in speech recognition systems that can generalize across various downstream tasks. However, despite recent advances in deep learning, transformer-based models remain understudied in the field of SER. By extracting feature embeddings from five variations —tiny, base, small, medium, and large —Whisper, an OpenAI transformer-based encoder-decoder model, is evaluated for SER. We evaluated the discriminative power of these embeddings using Logistic Regression(LogR), Multilayer Perceptron(MLP), Support Vector Machine(SVM), and Random Forest(RF) classifiers on two public benchmark datasets, German EMODB and English RAVDESS, which yielded promising results. We also combined Whisper embeddings with hand-crafted audio descriptors, including MFCC, mel spectrogram, chroma, spectral tonnentz, and zero-crossing rate, to further enhance performance. Using Whisper, medium and large embeddings outperformed existing standard approaches on both datasets. Utilizing five-fold cross-validation, Whisper large embeddings paired with a LogR classifier demonstrated an impressive average accuracy of 97.36%, while Whisper medium embeddings combined with hand-crafted features achieved the highest average accuracy of 97.74% on the EMODB dataset. For the RAVDESS dataset, integrating Whisper large embeddings with an MLP classifier yielded an accuracy of 91.67%, which improved slightly to 91.76% with combined features. On EMODB, Whisper large embeddings achieved an average accuracy of 93.87% using leave-one-speaker-out (LOSO) validation, and this increased to 94.79% with combined features using LogR. In the case of RAVDESS, the same setup attained an average accuracy of 79.72% with LogR and Whisper large embeddings. Among the classifiers, LogR and MLP emerged as the most effective, showing no statistically significant differences when evaluated through pairwise Nemenyi post-hoc comparisons. However, LogR yielded superior results in EMODB, whereas MLP performed better on RAVDESS in speaker-dependent settings. Although the system is efficient for deployment, it exhibits a notable latency bottleneck when utilizing higher-capacity Whisper models.