Enhancing One-Shot CT Image Segmentation with SAM and CLIP Prompt Learning

Recent advancements in artificial intelligence (AI) have witnessed the emergence of foundation models [5, 9] trained on extensive datasets across various domains. Particularly, SAM [5], initially designed for general image segmentation, has gained traction in medical imaging segmentation, with adaptations like SAM-Med2D [2] and MedSAM [7] fine-tuning SAM for medical tasks, while AutoSAM [3] and SAMed [10] modify SAM’s architecture for automated segmentation without prompts. However, automated segmentation without prompts hindered improvement, while fine-tuned SAM lacked automatic prompting. In the medical domain, where acquiring large annotated datasets is challenging, one-shot segmentation methods are crucial, allowing effective segmentation with minimal labeled data. This study explores one-shot medical image segmentation, leveraging SAM and fine-tuned CLIP Surgery, offering automated localization hints. Additionally, we introduce text prompts for SAM, bridging manual prompting and fully automated methods. Our method proposes a one-shot CT image segmentation pipeline integrating CLIP Surgery [6] and SAM, achieving effective segmentation using only annotated CT images from a single patient. We also test the pipeline’s potential with manual prompting. Experimental results show that our method outperforms state-of-the-art methods, particularly for large abdominal organs in one-shot scenarios, addressing the challenges of medical image segmentation effectively.