Prompt-SAM: A Vision-Language and SAM based Hybrid Framework for Prompt-Augmented Zero-Shot Segmentation

Abstract Recent advancements in deep learning have greatly improved computer vision tasks like object detection, image classification, and segmentation. Despite these successes, traditional supervised learning methods still depend on large annotated datasets, which are often expensive and time consuming to create. To overcome this limitation, we present a zero-shot segmentation framework that combines the strengths of CLIP (Contrastive Language-Image Pretraining), its segmentation-focused variant CLIPSeg, and the Segment Anything Model (SAM). This approach first uses the zero-shot classification ability of CLIP or CLIPSeg to produce initial segmentation cues. These cues, such as point and box prompts, are then refined by SAM to generate accurate segmentation masks. Using this prompt-based strategy, the system can perform segmentation without requiring labeled data, making it suitable for a wide range of domains, including both natural scenes and medical imaging. Our experiments on benchmarks such as MS-COCO, Pascal VOC, and chest X-ray datasets highlight the effectiveness of the method. In particular, the CLIPSeg+SAM combination achieves a mean IoU of 0.793 and a Dice score of 0.873 in the chest X-ray dataset, outperforming both CLIPSeg and SAM when used alone. Visual results also show that this method produces clearer and more precise mask boundaries, even in challenging or cluttered environments. In summary, the proposed training-free framework offers a scalable and generalizable solution for zero-shot segmentation, significantly reducing the reliance on annotated datasets while delivering strong performance on unseen classes.