Hybrid deep learning architecture for facial emotion recognition on FER2013 dataset

This work focuses on a reliable facial emotion recognition framework which aims to improve facial emotion classification performance on the FER2013 benchmark dataset. Recognition of facial emotions is now an important research field due to its uses in human computer interaction, healthcare, intelligent surveillance, education and affective computing. Despite the above, the human face still exhibits subtle changes with facial pose, facial illumination, occlusion and expression that continue to impact the overall recognition result. In order to solve these problems, this paper puts forward a hybrid deep learning architecture based on ResNet50 and Transformer encoder, capable of extracting multi-scale detailed spatial information and the relationships among various facial regions. An evaluation was performed with the proposed framework using the FER2013 database that contains 7 different emotion classes and 35,787 facial images. Images were pre-processed as three channel formats, resized to 224 × 224, normalized, and trained with the augmented image by flipping horizontally, random rotation, and resized cropping. For feature extraction, a pre-trained model was employed: the ResNet50 network. The Transformer encoder model was used to help enhance the contextual features representation, and the emotion classification was performed afterwards. The proposed framework has an overall accuracy of 89.7 percent on the official test partition (FER2013). Finally, an improvement was shown by ablation studies on the transformer where the ResNet50 model had a 72.84 percent, the ResNet50 model with a dense classifier reached 74.63 percent, and the complete framework achieved 89.7 percent. The results showed that the proposed method can be implemented transparently and reproduced and can robustly and consistently recognize facial emotions. The study provides a viable model which can guide the future research and practice of emotion recognition using facial expression.