Edge-SAN: An Edge-Prompted Foundation Model for Accurate Nuclei Instance Segmentation in Histology Images

Accurate nuclei segmentation is fundamental in histology image analysis, playing an essential role in cancer grading and diagnosis. However, this task remains challenging due to variations in staining protocols, heterogeneity among nuclei types, and the densely clustered nature of nuclei. While SAM exhibits zero-shot generalization capabilities in natural image segmentation, its performance degrades when applied to nuclei segmentation in histology images. Existing adaptations of SAM for medical imaging primarily focus on organ or lesion segmentation, which differs substantially from nuclei segmentation due to the unique characteristics of nuclei—specifically, their sparse distribution combined with dense clustering. To address these challenges, we propose Edge-SAN (Segment Any Nuclei with Edge Prompting), an interactive segmentation foundation model specifically designed for nuclei segmentation. Edge-SAN introduces a novel edge prompting method that enhances the delineation of nuclei boundaries, particularly among densely clustered nuclei, by leveraging edge information to improve segmentation accuracy. We evaluate Edge-SAN on 12 diverse datasets in both few-shot and zero-shot scenarios, demonstrating its effectiveness as a foundation model for nuclei segmentation, achieving 66.81% AJI and 73.13% DSC—improvements of 16.33% and 15.48% over SAM-Med2D, respectively. The code is available at https://github.com/deep-geo/Edge-SAN.