Convolutional Neural Network with Dual Penalty – Elastic Net and Federated Learning Method for Multimodal Emotion Recognition

Humans express their emotions in different ways, sparking interest in multimodal fusion-based emotion recognition, which uses multiple modalities to capture complete emotional data.However, extracting deep emotional features from various modalities and effectively fusing them remains challenging due to the presence of mixed classes in data.To mitigate this issue, in this article developed the Convolutional Neural Network (CNN) with Dual Penalty -Elastic Net (DP-ENet) regularization technique which includes for multimodal emotion recognition.The DP-ENet is incorporated in CNN model which improves the sparsity in every modality feature subspace by eliminating the noisy or irrelevant features.Additionally, it prevents large weight magnitudes in every modality of feature subspace, enhances smoothness, and improves generalization of the CNN model.Then, the Federated Learning (FL) technique is used for multimodal emotion recognition which preserves the privacy of the client's data by transferring the data from a local to a global server.The developed CNN with the DP-ENet method attained 93.27% accuracy on the IEMOCAP dataset and attained 94.53% accuracy on the CMU-MOSEI dataset.The CNN with DP-ENet method showed effective performance when compared to Cascaded Multi-Channel and Hierarchical Fusion (CMC-HF).