An Improved EfficientNet-B0 Model for Facial Emotion Recognition

Abstract Facial emotion recognition (FER) is increasingly built into interactive software, but reported accuracies are hard to compare because datasets, emotion sets and evaluation splits differ between studies, and few studies examine whether a larger backbone is worth its extra computational cost on the same data. This study develops and evaluates an EfficientNet-B0 model for five-class FER (happiness, sadness, anger, surprise and neutral) and compares it with EfficientNet-B1 and EfficientNet-B2 trained on the same images. FER2013 and the Real-world Affective Faces Database (RAF-DB) were combined, 1,470 duplicate or label-conflicting images and 19 unreadable files were removed, and a balanced set of 25,000 grayscale images (5,000 per class) was divided into 20,000 training, 2,500 validation and 2,500 test images, with no image shared between subsets. The model was fine-tuned in two stages from ImageNet weights using label smoothing, geometric and photometric augmentation, Mixup and CutMix. On the independent test set, EfficientNet-B0 achieved an accuracy of 78.68%, a macro F1-score of 78.83% and a macro-averaged ROC-AUC of 0.948, with class F1-scores ranging from 70.13% (neutral) to 87.78% (surprise). EfficientNet-B1 reached 79.76% accuracy, a difference that McNemar's test did not find significant (p = 0.107), while EfficientNet-B2, trained under a reduced schedule, reached 75.44% and was significantly less accurate (p < 0.001). EfficientNet-B0 used 36.6% fewer parameters than B1 (4.38 million against 6.90 million), needed 8.5 ms per image for single-image inference against 12.4 ms, required 40% less fine-tuning time and recorded the highest macro AUC of the three backbones. The trained model was deployed in a FastAPI web application that returns the predicted emotion with a confidence score. Within the conditions tested, EfficientNet-B0 gave the best balance between recognition performance and computational cost, while sadness and neutral remained the expressions most often confused with each other.