Multimodal Emotion Recognition Using Visual and Thermal Image Fusion: A Deep Learning Approach

Emotion recognition is becoming increasingly relevant in many domains, including healthcare, human-computer interaction, and affective computing. Accurate emotion recognition is essential for improving user experience, conducting mental health assessments, and developing adaptive systems. In our study, we provide a novel multimodal approach that combines visual and thermal data to improve emotion identification. Our system effectively extracts and merges features from both modalities by employing CNN architectures (VGG16, MobileNet, and ResNet50 for visual data and VGG16, MobileNet for thermal data). The fused feature set is classified using support vector machines (SVMs) and decision trees, with SVM accuracy of 97.70% and Decision Tree accuracy of 93.72%. Furthermore, SVM produced F1-scores of 97.31%, greatly exceeding the Decision Tree model with 88.40%. Class imbalance was reduced using strategic data augmentation, which improved the model’s performance even further. This study emphasizes the efficiency of multimodal systems by proving that mixing visual and thermal data improves the accuracy and resilience of emotion recognition. The suggested system’s versatility and efficiency make it an important tool for real-world applications including healthcare and human-computer interaction.