MRZegCLIP: Combine Multi-Scaled Region with ZegCLIP

With the remarkable zero-shot learning capabilities demonstrated by large vision and multimodal models, applying such models to zero-shot segmentation has become a mainstream research focus in this field. Among these methods, the ZegCLIP model enhances CLIP's image-level zero-shot classification capability by integrating visual prompts, relationship description modules, and a segmentation decoder, enabling pixel-level zero-shot segmentation with notable performance. However, although ZegCLIP achieves pixel-level segmentation, its decoder operates at the block level and relies on upsampling to produce pixel-level outputs from these block-based predictions. We propose that enhancing ZegCLIP's block-level classification capability through the use of multi-scale image blocks for training visual prompts and the segmentation decoder can provide richer visual information, thereby improving zero-shot segmentation performance. In this work, we introduce a method that selects multi-scale blocks from the input image to train both the visual prompts and segmentation decoder. Given that the introduction of multi-scale block cropping significantly reduces training efficiency, this paper employs a region proposal network to select blocks with greater training value, thereby significantly accelerating the training process. Experiments on the PASVOC 2012 Aug dataset show a 2.95% mIoU improvement, demonstrating that incorporating multi-scale image blocks enhances ZegCLIP's zero-shot segmentation capabilities.