Test-Time Medical Image Segmentation Using CLIP-Guided SAM Adaptation

Test-time medical image segmentation is a critical component in clinical practice, enabling pre-trained medical segmentation models to effectively adapt unseen medical samples with potential distribution shifts. However, existing methods are typically task-specific and restricted to certain diseases, with limited research focusing on test-time adaptation for universal segmentation, i.e. Segment Anything Model (SAM). Moreover, it is difficult to generate suitable prompts that can be effectively utilized by SAM for unseen test data without any label information. To address these challenges, we propose TTCS, a novel Test-Time medical image segmentation method using CLIP-guided SAM adaption, which achieves effective universal segmentation across diverse medical segmentation tasks. Specifically, we introduce a test-time prompt tuning strategy that leverages the semantic information from CLIP to generate precise prompts for each test data, effectively addressing the issue of poor prompt quality for SAM due to label scarcity. After generating the prompt for SAM, we implement an adaptive self-training strategy to further increase SAM’s robustness under distribution shifts. Our proposed method is inherently task-agnostic, and extensive experiments demonstrate the superior performance of our TTCS approach.