Speech Emotional Recognition using XGBoost and Deep Learning Algorithm and Multisource Datasets

Speech emotional recognition (SER) focuses on developing computers' comprehension and response to human emotional tones and is a key field of research in human-machine interaction. This paper outlines the implementation and evaluation of SER systems with machine learning algorithms and multisource datasets. To classify audio signals according to their emotional content, we use the XGBoost classifier, convolutional neural networks, and other deep learning algorithms. In this research, four datasets are used: CREMA-D, RAVDESS, SAVEE, and TESS. The process starts with audio data pre-processing, where we extract Mel Frequency Cepstral Coefficients (MFCCs) (specifically using Python's librosa library) to feed into our model. The XGBoost classifier first uses these features to establish a baseline performance. After that, deep learning models are applied, designed, and trained to improve the temporal and spatial characteristics inherent in speech signals for enhanced emotion recognition. A 1D CNN architecture, comprising convolutional layers, max pooling, batch normalization, and fully connected dense layers, is employed for feature extraction and classifies seven emotions: happy, sad, furious, neutral, fear, disgust, and surprise. The model was improved by using a categorical cross-entropy loss function, early stopping, and reducing the learning rate for regularization, all while using the Adam optimizer. The one that model was trained and evaluated using an 80-10-10 train-validation-test split, attaining approximately 90% accuracy on the test set and surpassing traditional machine learning models. The results were confirmed with a confusion matrix and classification reports, indicating the model's capability to recognize emotions from various datasets reliably. Finally concluded that the XGBoost classifier outperforms other algorithms, and the CREMA-D dataset is more suitable for audio emotion recognition. This work bears implications for AI emotion detection in real life, call centre sentiment analysis, healthcare monitoring, and personalized virtual assistants, thereby nurturing a paradigm shift toward more human and intelligent systems.