PG-HSAM: Prototype-Guided Hierarchical Segment Anything Model for Medical Image Segmentation with Limited Annotations

Abstract Adapting segmentation foundation models to medical images with limited annotations requires effective use of semantic information without substantially increasing model complexity. We propose PG-HSAM, an extension of H-SAM that introduces prototype prior fusion (PPF) and balanced pixel-to-prototype contrastive regularization (BPC). PPF maintains exponential-moving-average class prototypes, converts pixel-to-prototype similarities into a semantic probability prior, and fuses this prior with the first-stage prediction before hierarchical refinement. BPC regularizes the same feature space with class-balanced pixel sampling. For binary segmentation, foreground and background prototypes are jointly normalized to construct the semantic prior. The prototype memory is non-trainable, and BPC is used only during training. Under the limited-label protocols of H-SAM, PG-HSAM achieves mean Dice scores of 80.49%, 85.20%, and 83.51% on Synapse, LA, and PROMISE12, respectively. Compared with H-SAM, the gains are 0.81, 1.26, and 6.56 percentage points, while Synapse HD95 decreases from 18.35 to 13.70. Ablation and hyperparameter studies further verify the contributions of PPF and BPC and show that the improvement is obtained without increasing the model parameter count.