Development of Emotion Recognition via Federated Learning: A Multi-modal Approach with Hybrid Convolutional Network

Human emotion recognition is the crucial process for performing effective communication and intelligent interaction. Identifying human emotion through discrete technologies is an essential step. Affective computing is one machine learning process in which multi-task learning has gathered a great deal of attention. It leverages shared data across associated activities to enhance performance. The physiological results from the aforementioned devices may also constitute a privacy risk because they reveal extremely sensitive information. To address this issue, we develop a federated learning approach with multi-task learning to precisely recognize facial emotion from the human. The required face images and Electroencephalogram (EEG) signals are collected from online sources. The face images are considered as feature set 1. From the EEG signals, the deep features are extracted through the Restricted Boltzmann Machine (RBM) and these features are considered as feature set 2. Both feature set 1 and feature set 2 are applied to the federated learning network, where the Hybrid Convolution-based Convolutional Neural Network (HC-CNN) is adopted for the recognition of human facial emotions. The face images are applied to the 2D layer and the extracted deep features from the EEG signal are applied to the 1D layer of the hybrid convolution network to provide an effective face recognition outcome. The recognition outcome of the proposed model is compared with the traditional face emotion recognition models concerning various measures. The recommended model successfully recognizes the emotion of the human with an accuracy of ${9 5. 7 5 \%}$ and precision of ${9 5. 9 7 \%}$, respectively.