Semi-Supervised Facial Emotion Recognition via Valence-Arousal Pseudo-Label Refinement

Facial expression recognition is a pivotal area in computer vision, traditionally focusing on categorical labels such as ‘Happy’, and ‘Sad’. Recent advancements have transitioned towards using the continuous indicators valence and arousal, reflecting the complicated nature of human emotions. This study introduces a method using semi-supervised learning to generate valence and arousal labels for existing categorical datasets, addressing challenges like racial bias. We propose a pseudo-label refinement framework, VAP-Refine (Valence-Arousal Pseudo-label Refinement), which enhances facial expression recognition by combining teacher model predictions and category-level statistics. These predicted labels are adjusted using ground truth category information and combined with actual category labels to train a more robust facial expression recognition model. Our approach improved accuracy from 69.3% to 72.26% and from 62.94% to 67.24% across two datasets. Fine-tuning with a teacher model predicting valence and arousal achieved 75.82% and 91.58% accuracy, with an F1-score of 0.9147, despite data imbalances. These results highlight the potential of semi-supervised learning to enhance facial expression recognition by incorporating continuous emotional indicators, improving model performance and contributing to more accurate affective computing applications. Furthermore, the proposed framework consistently improved performance across various backbone architectures, including ResNet50, SHViT, and DDAMFN++, highlighting its generalizability and versatility.