Masked Contrastive Representation Learning for Self-Supervised Visual Pre-Training

Self-supervised learning has achieved state-of-the-art performance in various tasks and applications. In computer vision, self-supervised learning often employs contrastive learning and masked image modeling, each with its limitations: contrastive learning heavily relies on strong data augmentation and large batch sizes, etc., while masked image modeling struggles to capture high-level semantics and discrimination. In this work, we introduce MAsked Contrastive Representation Learning (MACRL), a novel framework that integrates both paradigms through an asymmetric siamese network design. The online and momentum branches of the network receive asymmetric data augmentation operations and extract features through their encoders. The decoder in the online branch reconstructs the original image, while the projectors in both branches compute the contrastive loss. The online branch and the momentum branch are updated through gradient backpropagation and exponential moving average, respectively. MACRL jointly optimizes the reconstruction and the contrastive objectives to encourage representations with enhanced discrimination and semantics. Experimental results show that MACRL achieves competitive performance in downstream vision tasks, including image classification and semantic segmentation. Moreover, it demonstrates consistent performance across both large-scale and small-scale datasets.