EMO-FlexNet: A Domain-Aware and Explainable Multimodal Fusion Framework for Emotion Recognition

Emotion recognition plays a vital role in affective computing but faces challenges like noisy data, incomplete feature extraction, and limited cross-modal adaptability. Existing multimodal models often struggle to balance accuracy, efficiency, and interpretability across diverse real-world scenarios. This paper proposed an advanced multimodal emotion recognition framework, Emotion-Aware Flexible Network for Multimodal Fusion (EMO-FlexNet), designed for efficiency, interpretability, and domain-awareness. A domain-specific BERT model is pretrained on emotion-centric corpora to better capture affective cues in text. For audio, a refined Audio Spectrogram Transformer (AST) is employed, enhanced through emotion-domain pretraining, multi-scale spectrogram input, and fusion of prosodic features. Visual features are extracted using a Depthwise Separable Convolution-based ResNet, significantly reducing computational overhead while maintaining expressive power for facial emotion representation. Cross-modal attention mechanisms are used to capture interdependencies across modalities, and Multimodal Compact Bilinear Pooling (MCB) facilitates efficient feature fusion. To promote model transparency, SHAP and Grad-CAM are incorporated for explainability. To optimize hyperparameters and fusion weights effectively, the Adaptive Cheetah Optimization Algorithm (AChOA) is integrated, enhancing model convergence and robustness. AChOA performance is compared against existing metaheuristics such as Crocodile Optimization Algorithm (CrOA), Grey Wolf Optimization (GWO), Particle Swarm Optimization (PSO), and Ant Colony Optimization (ACO). Comparative evaluations on benchmark MELD datasets demonstrate that the proposed EMO-FlexNet outperforms state-of-the-art methods, including TRAVENet, MultiEmotionNet, EfficientNet, and TCN, in accuracy, robustness, and interpretability, making it ideal for affective computing and empathetic AI applications.