Synthetic data generation for modality-agnostic zero-shot promptable medical image segmentation

Foundation models for image segmentation have recently surged in popularity after the advent of the Segment Anything Model (SAM). While SAM demonstrates remarkable zero-shot segmentation capabilities for natural images, it struggles to achieve satisfactory performance on medical images. This can be attributed to the significant differences in image features such as textures, noise patterns, and contrast differences between medical and natural images. Although re-training SAM on medical image data can potentially improve its performance, it is extremely time-consuming and costly to acquire annotations for medical images. Hence, we propose a comprehensive analytical synthetic data simulation strategy that emulates the complexity of medical imaging with high fidelity. This simulation method captures essential elements of medical imaging, including a range of anatomical shapes, differences in contrast and texture, and the placement of important structures near surrounding anatomical features. We used this data to fine-tune the pre-trained SAM model and evaluated its segmentation performance using a single positive prompt on eight anatomical structures of CT and MRI modalities acquired from five publicly available datasets. Our model achieved superior zero- shot performance without the additional cost of training data collection and annotation, demonstrating the effectiveness of the proposed data synthesis strategy. The fine-tuned weights can also provide researchers a better starting point for medical image segmentation, allowing them to use our model out-of-the-box or fine-tune it for specific imaging modalities.