EfficientNet Convolutional Neural Network with Gram Matrices Modules for Predicting Sadness Emotion

Images are becoming an attractive area of emotional analysis. Recognising emotions in the images of general nature is gaining more and more research attention. Such emotion recognition is more sophisticated and different from conventional computer tasks. Due to human subjectivity, ambiguous judgments, cultural and personal differences, there is no an unambiguous model for such emotion assessment. In this paper, we have chosen sadness as the main emotion, which has significant impact to the richness of human experience and the depth of personal meaning. The main hypothesis of our research is that by extending the capabilities of convolutional neural networks to integrate both deep and shallow layer feature maps, it is possible to improve the detection of sadness emotion in images. We have suggested integration of the different convolutional layers by taking the learned features from the selected layers and applying a pairwise operation to compute the Gram matrices of feature sub-maps. Our findings show that this approach improves the network’s ability to recognize sadness in the context of binary classification, resulting in a higher emotion recognition accuracy. We experimentally evaluated the proposed network for the stated binary classification problem under different parameters and datasets. The results demonstrate that the improved network achieves improved accuracy as compared to the baseline (EfficientNetV2) and the previous state-of-the-art model.