Affinity Similarity-Based Contrastive Loss for Unsupervised Visual Representation Learning

Contrastive loss and its variants are very popular for visual representation learning in an unsupervised scenario, where positive and negative pairs are produced to train a feature encoder based on data augmentation. However, treating only the original image and its augmented versions as positive pairs may lead to problems, since two training images that share the same object category may be misclassified as negative pairs. To address this issue, this paper introduces a plug-in-play affinity similarity module (ASM) used in any contrastive learning framework for unsupervised visual representation learning. In this approach, positive and negative pairs are determined not only based on data augmentation, but also according to the similarity of their feature embeddings. The core idea is that two training samples with higher feature similarities are likely to belong to the same object category, suggesting that they should be classified as a positive pair with high probability, and vice versa for negative pairs. Accordingly, we have also developed an improved affinity similarity-based contrastive loss (ASCL) that leverages the newly generated positive and negative training pairs produced by the ASM. To demonstrate the effectiveness of our method, we extensively evaluate its performance on the ImageNet dataset, showing that it achieves satisfactory results.