Constructing multi-modal emotion recognition model based on convolutional neural network

As society advances, an increasing number of individuals spend significant time interacting with computers daily. To enhance the human-computer interaction experience, it has become crucial to augment the computer’s ability for emotion recognition. This capability holds excellent importance as machines become capable of responding to us in a more natural and contextually relevant manner, aligned with our current emotional states. Examples of such applications include caregiving and social robots. Accurate recognition of human emotions, followed by the ability to determine the most appropriate responses, can significantly enhance user experiences. The most commonly employed methods in emotion recognition include observing facial expressions, audio, and conversational content. The multi-modal emotion recognition lacks the explicit mapping relation between emotion state and audio and image features. This study proposes a fusion method for audio-visual emotion recognition. The audio and video data are preprocessed separately. The audio emotion features and visual expression features were then extracted using two distinct feature extractors. The audio emotion feature extractor, denoted as audio-net, employs a 2D CNN architecture capable of processing image-based Mel-spectrograms as input data. The facial expression feature extractor, visual-net, uses a 3D CNN architecture to process sequences of facial expression images. Fusing the visual and auditory features and enhancing feature correlation using the deep canonical correlation analysis (DCCA) method. This research uses the eNTERFACE05 dataset and reaches 89.13% accuracy in classifying emotions. The result shows that considering audio and facial features at the same time can the model better recognize the emotion people are having.