HAT-SAM3: endoscopy-aware adaptation of a foundation segmentation model for generalizable polyp segmentation

Abstract Automatic polyp segmentation remains difficult when models are evaluated outside their training distribution. Polyps vary substantially in scale, shape, color, boundary contrast, and surface appearance; colonoscopy further introduces specular reflection, blur, fluid, and illumination change. These factors make cross-dataset generalization a stricter problem than in-domain accuracy. In this paper, we present HAT-SAM3, an endoscopy-aware foundation model adaptation based on highlight-aware training (HAT) for generalizable polyp segmentation. Instead of designing another task-specific decoder, we adapt a visual foundation model, SAM 3, as a pretrained segmentation prior for cross-dataset polyp segmentation. To account for unreliable local appearance cues caused by specular saturation, we use a training-only boundary regularizer that stochastically adds saturated pixels in a narrow exterior band around annotated polyp masks to the foreground supervision, while leaving inference unchanged. In the primary five-dataset public benchmark, HAT-SAM3 achieves an average mDice of 0.884 and the best listed Dice on four of five datasets, with the most consistent improvements on out-of-domain test sets. In an additional Kvasir-to-external zero-shot evaluation, HAT-SAM3 improves mDice over the strongest reported baseline by 0.062–0.131 across three external benchmarks. Together, these evaluations indicate that endoscopy-aware adaptation of a foundation segmentation prior can improve cross-dataset polyp segmentation. The code is publicly available at https://github.com/HaoLi12345/polyp .