Staged Transfer Learning for Multi-Label Facial Emotion Recognition from Full Faces

Human emotions play a crucial role in shaping human decisions and actions. These emotions can be conveyed through various channels, including facial expressions, vocal inflections, body language, written communication, and physiological changes. Affective computing and emotion recognition aim to develop software systems that can detect and classify human emotions. Researchers have traditionally employed two main approaches: computer vision and deep learning. Facial emotion recognition focuses on categorizing human emotions based on facial image data. While prior research has primarily focused on recognizing single emotions (multiclass), this study addresses the challenge of multilabel recognition, where an image can depict multiple emotions simultaneously. To address this complexity, we introduce a novel deep learning technique called staged transfer learning. This approach breaks down the learning process into manageable subtasks, where each stage leverages the knowledge acquired from the previous stage. We utilized the unique “emoface” dataset containing multi-label facial imagery to evaluate the effectiveness of our proposed method. We implemented a two-step deep learning approach for facial emotion recognition. This approach leverages both custom convolutional neural networks (CNNs) and pre-trained models via transfer learning. In the first stage, transfer learning was applied to the fer2013 dataset, a large multiclass collection, to extract robust features for recognizing seven basic emotions (happiness, sadness, surprise, fear, anger, and disgust). Subsequently, the second stage transferred this learned knowledge to the emoface dataset, which is a smaller multilabel dataset encompassing 25 emotions. This staged approach achieved high performance, with an average binary accuracy of 0.9389 (training), 0.9220 (validation), and 0.9187 (testing), along with low binary cross-entropy loss. These results outperform the baseline models and existing research on multi-label facial emotion recognition, validating the effectiveness of our proposed method. This ability to recognize multiple emotions simultaneously opens doors for applications in various fields, including affective computing, healthcare, human-computer interaction, robotics, and potentially autonomous vehicles.