Zoom-shot: Fast, efficient and unsupervised zero-shot knowledge transfer from CLIP to vision encoders

• Efficiently transfers zero-shot capabilities from CLIP to pre-trained vision encoders • Improves performance over previous SOTA work in domain • Training data diversity/coverage improves mapping quality and zero-shot performance • Increased data coverage achieved through the use of tailored loss functions • Training data is entirely unlabelled and unpaired image and text data Foundation models like CLIP demonstrate exceptional capabilities over a broad domain of knowledge, such as with zero-shot classification; however, they also require significant computational resources, narrowing their real-world utility. Recent studies have shown that mapping features from pre-trained vision encoders into CLIP’s latent space can transfer some of CLIP’s abilities to smaller vision encoders, offering a promising alternative. Yet, the performance of these vision encoders still falls short of CLIP’s native capabilities, particularly in low-data regimes. In this work, we argue that enhancing training data coverage/diversity significantly improves mapping efficacy. We achieve this using tailored loss functions rather than relying on data augmentation or increasing training samples. For instance, we exploit the inherent multimodal nature of CLIP’s latent space, by incorporating cycle-consistency loss as one of our loss functions. Moreover, the mapping is learned using entirely unlabelled and unpaired data, eliminating the need for manual labelling or data pairing in novel domains. From these findings, our resulting method (Zoom-shot) offers a viable path to flexible zero-shot models for resource-limited, data-scarce settings. We test Zoom-shot’s zero-shot performance across various pre-trained vision encoders on coarse- and fine-grained datasets and achieve superior performance compared to recent works. In our ablations, we find Zoom-shot allows for a trade-off between data and compute during training; allowing for a significant reduction in required training data. All code and models are available on GitHub.