MULTIMODAL DEEP LEARNING FOR EMOTION RECOGNITION: A FUSION OF EEG, FACIAL EXPRESSIONS AND VOICE IN HUMAN-COMPUTER INTERACTION

<p>Emotion recognition systems have become increasingly critical for developing emotionally intelligent humancomputer interfaces, yet existing single-modal approaches suffer from significant limitations in real-world<br>deployment scenarios. This research presents a comprehensive multimodal deep learning framework that<br>synergistically integrates electroencephalography (EEG), facial expressions, and voice signals to achieve<br>robust emotion recognition across diverse conditions. We introduce the Multimodal Emotion Recognition<br>Dataset (MERD-2025), a novel synchronized dataset comprising recordings from 150 demographically diverse<br>participants across ten distinct emotion categories, providing unprecedented data quality and cultural<br>representation for emotion recognition research.<br>Our proposed Attention-based Multimodal Fusion Network (AMF-Net) employs specialized encoders for each<br>modality, coupled with a novel cross-modal attention mechanism that dynamically weights modality<br>contributions based on emotional context and signal reliability. The architecture processes 64-channel EEG<br>signals, high-resolution facial videos, and audio recordings through modality-specific neural networks before<br>fusing representations through learned attention weights. Extensive experimental validation demonstrates that<br>AMF-Net achieves state-of-the-art performance with 87.3% classification accuracy, representing substantial<br>improvements of 8-15% over single-modal approaches and 4-6% over traditional fusion methods.<br>Comprehensive analysis reveals consistent performance across demographic groups, robust handling of<br>missing modalities, and interpretable attention patterns that provide insights into emotion-specific modality<br>contributions. The system maintains practical computational requirements while demonstrating resilience to<br>noise and incomplete data conditions. These findings establish significant potential for real-world applications<br>in adaptive user interfaces, mental health monitoring, educational technology, and automotive safety systems.<br>The research advances the field by providing both technological innovations and comprehensive datasets for<br>future multimodal emotion recognition development.<br>Keywords: Emotion recognition, Multimodal fusion, Deep learning, EEG, Facial expressions</p>