Performance Evaluation of Different Speech-Based Emotional Stress Level Detection Approaches

This paper presents a comprehensive evaluation and comparison of selected conventional and advanced approaches for detecting and classifying emotional stress levels from speech. Three representative and publicly available stress datasets – CRISIS, StressDat, and SUSAS – were utilized. A detailed comparative analysis of their speech characteristics, including fundamental frequency (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub>), formant frequencies (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub>, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub>, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i><sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sub>), voice intensity, harmonic-to-noise ratio, jitter, and shimmer, was conducted to investigate how these features change with increasing stress and contribute to its detection. Both conventional feature-based methods and deep learning techniques, including transfer and self-supervised learning, are explored in the experimental part of this research. The best classification results among the conventional methods were achieved using a combination of features extracted via the Discrete Wavelet Transform and GammaTone Cepstral Coefficients (DWT+GTCC), paired with a Subspace <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i>-Nearest Neighbors classifier, which yielded F1-scores of up to 91.1% on CRISIS and 82.4% on StressDat. Among the deep learning approaches, we fine-tuned self-supervised models such as Wav2Vec 2.0 and BYOL-S (Bootstrap Your Own Latent for Speech), as well as transfer learning models including VGGish and YAMNet.Wav2Vec 2.0 consistently outperformed the others, achieving an F1-score of 94% on CRISIS and over 87% on StressDat. On the SUSAS dataset, which contains speech under simulated and real stress conditions, an F1-score of 77% was achieved using both the fine-tuned Wav2Vec 2.0 model as well as a pipeline combining BYOL-S feature extraction with a Support Vector Machine (SVM) classifier. After merging all stress-related datasets into a single balanced dataset,Wav2Vec 2.0 achieved the highest F1-score of 85%. Incorporating additional emotional speech data improved robustness and led to a slight increase in the accuracy of classifying stress into three distinct levels. These results highlight the importance of robust feature selection, appropriate classifier design, and the integration of emotional speech data to enhance generalization performance in real-world applications.