Unleashing the Potential of Generalist Segmentation Foundation Models for Biomedical Image and Video Analysis

ABSTRACT The unprecedented developments in generalist segmentation foundation models have become a dominant focus in the field of computer vision, introducing a multitude of previously unexplored capabilities in a wide range of natural image and video analysis tasks. From the pioneering segment anything model (SAM) that revolutionized prompt‐driven image segmentation to the recent SAM2 which enables streaming video with robust spatiotemporal consistency, these models have demonstrated effective adaptability in natural scenarios and show strong potential for biomedical applications. In this paper, we present a comprehensive and in‐depth review of the development, adaptation, and application of generalist segmentation foundation models in biomedical domains. We first contextualize the evolution of key models and their core mechanisms, highlighting their potential for bridging the gap between general vision and specialized biomedical tasks. We then systematically examine the challenges in applying these models to biomedical data, including domain shift, ambiguous boundaries, and dimensional gaps for 3D medical images. Finally, we articulate our perspectives on the future research directions. This review aims to provide a roadmap for researchers, facilitating the translation of generalist segmentation capabilities into effective biomedical solutions.