Analysis and Experimental Study of Zero-Shot Capabilities in Vision-Language Models: An In-Depth Exploration of Contrastive and Masked Methods

This paper conducts a thorough analysis and experimental evaluation of zero-shot capabilities in vision-language models (VLMs), concentrating on three distinct approaches: contrastive learning, masked learning, and generative modeling, exemplified by CLIP, FLAVA, and CoCa, respectively. CLIP uses contrastive learning to align images and text robustly, FLAVA employs masked learning to improve multimodal reasoning, and CoCa combines generative captioning with contrastive learning for fine-grained multimodal comprehension. Zero-shot learning, a pivotal AI capability, allows models to apply knowledge to new tasks without further training specific to those tasks. The performance of these models is tested through experiments in zero-shot settings, including image classification on datasets like CIFAR-100, Flowers102, and Food101, to evaluate generalization to new image categories. Furthermore, zero-shot image and text retrieval tasks are performed using Flickr30k and MSCOCO benchmarks to measure the models' ability to align and retrieve across modalities without direct supervision. Results from these tests provide a comprehensive look at the VLMs' zero-shot performance, highlighting their potential and limitations in real-world applications on unseen data.