Cluster-SCP: Similarity and Contrastive Learning to Enhance Pseudo Labels for Fine Tuning under Few Labels

Language models often underperform when fine tuned with limited labelled data. To reduce dependence on costly annotation, recent studies have explored clustering and pseudo labelling to leverage unlabeled text with only a few labelled examples for fine tuning. However, these methods face persistent challenges: clustering errors and pseudo-label mismatches can degrade fine-tuning performance. We propose Cluster-SCP, a novel framework that improves intra-cluster coherence and reduces cluster quantity to generate more reliable pseudo labels. The framework begins with K-Means initialization and applies an iterative refinement process, Intermediate Pseudo Labels, comprising two stages: embedding reassignment to minimize clustering errors, and cluster merging via contrastive learning with graph readouts. The refined pseudo labels are first used to fine tune language model, which is subsequently refined using a small set of labels. Experiments on three benchmark datasets show that Cluster-SCP consistently outperforms baseline and state-of-the-art methods, achieving at least a 5.3% accuracy improvement while reducing the number of clusters by 22%.