Generalist models in medical image segmentation: A survey and performance comparison with task-specific approaches

Following the successful paradigm shift of large language models, which leverages pre-training on a massive corpus of data and fine-tuning on various downstream tasks, generalist models have made their foray into computer vision. The introduction of the Segment Anything Model (SAM) marked a milestone in the segmentation of natural images, inspiring the design of numerous architectures for medical image segmentation. In this survey, we offer a comprehensive and in-depth investigation of generalist models for medical image segmentation. We begin with an introduction to the fundamental concepts that underpin their development. Then, we provide a taxonomy based on features fusion on the different declinations of SAM in terms of zero-shot, few-shot, fine-tuning, adapters, on SAM2, on other innovative models trained on images alone, and others trained on both text and images. We thoroughly analyze their performances at the level of both primary research and best-in-literature, followed by a rigorous comparison with the state-of-the-art task-specific models. We emphasize the need to address challenges in terms of compliance with regulatory frameworks, privacy and security laws, budget, and trustworthy artificial intelligence (AI). Finally, we share our perspective on future directions concerning synthetic data, early fusion, lessons learnt from generalist models in natural language processing, agentic AI, and physical AI, and clinical translation. We publicly release a database-backed interactive app with all survey data ( https://hal9000-lab.github.io/GMMIS-Survey/ ). • Taxonomy on generalist models for medical image segmentation. • Architectural dissection of the most advanced generalist models. • Performance comparison between generalist and state-of-the-art task-specific models. • Comprehensive discussion on challenges and future directions on generalist models. • Interactive web-app for quick organ-wise and dataset-wise results visualization.