ECL-ConvNeXt: An Ensemble Strategy Combining ConvNeXt and Contrastive Learning for Facial Beauty Prediction

Facial beauty prediction (FBP) is a cutting-edge topic in deep learning, aiming to endow computers with human-like esthetic judgment capabilities. Current facial beauty datasets are characterized by multi-class classification and imbalanced sample distributions. Most FBP methods focus on improving accuracy (ACC) as their primary goal, aiming to indirectly optimize other metrics. In contrast to ACC, which is well known to be a poor metric in cases of highly imbalanced datasets, the recall measures the proportion of correctly identified samples for each class, effectively evaluating classification performance across all classes without being affected by sample imbalances, thereby providing a fairer assessment of minority class performance. Therefore, targeting recall improvement facilitates balanced classification across all classes. The Macro Recall (MR), which averages the recall of all the classes, serves as a comprehensive metric for evaluating a model’s performance. Among numerous classic models, ConvNeXt, which integrates the designs of the Swin Transformer and ResNet, performs exceptionally well regarding its MR but still suffers from inter-class confusion in certain categories. To address this issue, this paper introduces contrastive learning (CL) to enhance the class separability by optimizing feature representations and reducing confusion. However, directly applying CL to all the classes may degrade the performance for high-recall categories. To this end, we propose using an ensemble strategy, ECL-ConvNeXt: First, ConvNeXt is used for multi-class prediction on the whole of dataset A to identify the most confused class pairs. Second, samples predicted to belong to these class pairs are extracted from the multi-class results to form dataset B. Third, true samples of these class pairs are extracted from dataset A to form dataset C, and CL is applied to improve their separability, training a dedicated auxiliary binary classifier (ConvNeXtCL-ABC) based on ConvNeXt. Subsequently, ConvNeXtCL-ABC is used to reclassify dataset B. Finally, the predictions of ConvNeXtCL-ABC replace the corresponding class predictions of ConvNeXt, while preserving the high recall performance for the other classes. The experimental results demonstrate that ECL-ConvNeXt significantly improves the classification performance for confused class pairs while maintaining strong performance for high-recall classes. On the LSAFBD dataset, it achieves 72.09% ACC and 75.43% MR; on the MEBeauty dataset, 73.23% ACC and 67.50% MR; on the HotOrNot dataset, 62.62% ACC and 49.29% MR. The approach is also generalizable to other multi-class imbalanced data scenarios.