AAT-CGF: A Cross-Modal Deep Fusion Framework with Attention Aggregation and Cross Graph Fusion for Multimodal Emotion Recognition

Multimodal emotion recognition plays a crucial role in advancing human-computer interaction, with applications in mental health, education, and intelligent systems. However, effectively capturing the dynamic and complementary interactions among heterogeneous modalities remains a significant challenge. In this paper, we propose a novel framework, AAT-CGF, which integrates Attention-based Aggregation and Cross-modal Graph Fusion to enhance the modeling of fine-grained emotions across audio, visual, and textual modalities. The framework first extracts modality-specific features using LSTM networks for audio and visual data, and BERT for text. To capture subtle emotional cues, fine-grained features are further refined via dynamic time warping and facial landmark detection. These features are aggregated using a self-attention mechanism and subsequently fused through a graph-based structure that dynamically learns cross-modal relationships. Extensive experiments on benchmark datasets, including CMU-MOSI, IEMOCAP, and CASIA, show that AAT-CGF achieves competitive performance across various evaluation metrics, demonstrating its effectiveness in capturing multimodal emotional information. The proposed approach contributes a promising and scalable solution for advancing multimodal emotion recognition.