Enhancing Emotion Recognition Through Federated Learning: A Dual Neural Network Approach and Firefly Algorithm Optimization

Emotion recognition using multimodal methods, i.e. integrating speech and facial expressions, has garnered considerable interest due to its potential to enhance human–computer interaction, healthcare, and mental health monitoring. Accurate emotion recognition from both auditory and visual inputs is essential for creating intelligent systems that can empathetically interact with users. Furthermore, Federated Learning (FL) has also proven to be an important method for decentralized model training, providing privacy-preserving abilities at the cost of slightly lower accuracy. The goal of this study is to build a privacy-preserving, multimodal emotion recognition system combining speech and facial expressions with FL to enhance emotion detection performance with data privacy protection across distributed clients. This research employs the RAVDESS dataset in speech emotion recognition with a Long Short-Term Memory (LSTM) model for interpreting spoken patterns. Facial expression recognition makes use of the 3D Facial Expression Recognition Data dataset by utilizing a 3D Convolutional Neural Network (3D-CNN) for different facial emotions’ classification. These models are developed with FL, a distributed form, so that data is stored locally in the devices. For hyperparameter tuning, the Firefly Algorithm (FA) is used to optimize both models’ performances. A multimodal approach, where LSTM is used for speech recognition and 3D-CNN is used for facial expression recognition, achieved 99.6% accuracy on the RAVDESS dataset and 99.4% accuracy on the 3D Facial Expression Recognition Data in the proposed framework. The federated model performed with an F1-score of 96.8%, a precision of 96.3%, and a recall of 95.4%. The performance shows the excellence of fusing speech and facial expression recognition in a FL environment in an E-Learning Application. Future work can be applied to real-world applications in healthcare and interactive systems.