CLIP-assisted Pseudo-Label Enhanced Semi-Supervised Multi-Label Learning

Semi-Supervised multi-label learning (SSMLL) can handle multi-label scenarios with only a small amount of labeled data. It generates pseudo-labels for unlabeled samples to leverage the information hidden within them. Existing methods mainly focus on estimating the true number of pseudo-labels but tend to overlook the direct impact of model predictions on the quality of these pseudo-labels. These methods do not explicitly improve prediction accuracy, leading to low-quality pseudo-labels and a decline in student model performance. Therefore, improving pseudo-label quality in SSMLL is a key challenge. In this paper, we propose CLAPS (CLIP-Assisted Pseudo-Labeling Strategy) to address this issue. One reason for the poor quality of pseudo-labels in existing class-aware methods is the teacher model’s performance limitation during the early training stage. We leverage CLIP’s strong visual-text matching capability to assist in generating initial pseudo-labels. Specifically, we generate feature vectors from both coarse-grained and fine-grained perspectives for each image. Pseudo-Labels are then generated by measuring the similarity between text features and both types of image features. This approach effectively mitigates the problem of low-quality pseudo-labels during the early training stage. Additionally, we propose a dynamic pseudo-label update method based on adversarial noise, which helps reduce the impact of noisy labels during further training. Experimental results show that CLAPS improves the reliability of pseudo-labels, better captures the dependencies between multi-labels, and enhances the model’s robustness to noisy labels. It also achieves noticeable improvements compared to the existing class-aware method CAP.