Deep Multimodal Emotion Recognition: Fusion of Facial Features and Neurophysiological Signals

Multimodal emotion recognition through the fusion of facial and neurophysiological features plays an important role in various applications, such as advertising, the automotive industry, wearable devices, and human-computer interactions. Fusing human facial expressions and neurophysiological signals traditionally requires domain-specific knowledge and complex preprocessing steps. However, with the advent of deep learning, we can fully leverage the end-to-end capabilities of these techniques for the intermediate integration of facial and neurophysiological signals in emotion recognition systems. As a result, we introduce a novel end-to-end deep network that leverages transformers to learn rich feature representations of neurophysiological signals, integrated with a transformer-inspired technique for facial expression recognition and emotion classification. By integrating transformers and deep neural networks, our approach successfully captures complex temporal and spatial patterns in the data. This combination allows for more robust analysis, enhancing the system's overall performance in recognizing and classifying emotions accurately. We validated our approach through experiments on the well-known DEAP dataset, achieving performance comparable to the state-of-the-art, with accuracy rates of 97.64% for valence and 97.78% for arousal.