A HYBRID DATA MERGING AND AUGMENTATION-BASED DEEP FRAMEWORK FOR NEGATIVE AFFECT DETECTION

Emotion Recognition (ER) refers to capturing the different aspects pertaining to detection of emotions. Performance of ER models is hindered by several challenges including lack of generalization. The latter is influenced by factors including class imbalance, intra-domain differences, occlusions, varied image sizes and dimensionality of datasets. While Data Augmentation (DA) remains a prevailing solution, there is a pressing need for large, balanced, and heterogeneous datasets representing diverse real-world scenarios. Additionally, there is lack of comprehensive framework to combine augmentation and merging, particularly for applications in mental health. To address these issues, we propose a Recommendations and Behavioural support-based Framework for Negative Affect detection (RBF-NA). At the core of RBF-NA, we introduce an interesting pipeline, the Hybrid-Multi Stage Augmentation and Merging (H-MSAM). This pipeline is supported by introducing a novel algorithm AG-L1. Furthermore, a custom deep model for emotion detection, the Convolutional-Face Emotion Detection (C-FED) is proposed. C-FED achieves exceptional performance with highest accuracies of 99.75% on CK+ and 85.87% on KDEF datasets. H-MSAM improves the accuracy by 13.31%, 16.96%, and 19.44% on CK+, KDEF, and FER2013, respectively. Furthermore, C-FED outperforms ResNet and VGG16, with accuracy gains of 6.87% and 5.13%, respectively. The granular performance of system demonstrates excellent performance on every emotion class, with the negative emotions including sad, angry achieving superior results. The promising findings in this paper highlight the potential of RBF-NA in advancing ER research and fostering early detection of negative emotions.