Exploring joint embedding predictive architectures for pretraining convolutional neural networks

Joint Embedding Predictive Architectures present a novel intermediate approach to visual self-supervised learning combining mechanisms from instance discrimination and masked modeling. CNN-JEPA adapts this approach to convolutional neural networks and demonstrates its computational efficiency and accuracy on image classification benchmarks. In this study, we investigate CNN-JEPA, adapt it for semantic segmentation, and propose a learning objective that improves image-level representation learning through a joint embedding predictive architecture. We conduct an extensive evaluation, comparing it with other SSL methods by analyzing data efficiency and computational demands across downstream classification and segmentation benchmarks. Our results show that its classification and segmentation accuracy outperforms similar masked modeling methods such as I-JEPA and SparK with a ResNet-50 or a similarly sized ViT-Small encoder. Furthermore, CNN-JEPA requires fewer computational resources during pretraining, demonstrates excellent data efficiency in data-limited downstream segmentation, and achieves competitive accuracy with successful instance discrimination-based SSL methods for pretraining encoders on ImageNet. • We elaborate on CNN-JEPA’s performance on image classification and segmentation tasks. • We propose a global learning objective to enhance classification accuracy. • CNN-JEPA outperforms similar masked modeling self-supervised learning (SSL) methods. • It achieves accuracy competitive with top instance discrimination-based SSL methods. • It is faster and uses a simpler architecture compared to state-of-the-art SSL methods.