DSAM: A Faster SAM for 3D Medical Image Segmentation

Segment Anything Models (SAMs) have gained sig-nificant attention for their impressive zero-shot generalization capabilities. However, to effectively apply SAMs to 3D medical image segmentation tasks, rapid inference is essential. The main challenges currently include high memory demands and lengthy processing times. In particular, for volumetric images, 2D SAMs require repetitive computations across each slice, while 3D SAMs face exponential increases in model param-eters and floating-point operations. To address the time and memory constraints of applying SAMs to volumetric images, we propose Distilled SAM (DSAM) model, which achieves a 22-fold speedup compared to SAM when processing 128 x 128 x 128 volumetric images on a 2080ti GPU. The proposed DSAM is driven by two key innovations: 1) A novel distillation method that transfers knowledge from a 12-layer teacher network to a 6-layer student network; 2) Optimization of the original SAM's decoder, significantly reducing memory usage and computational time. Experiments on the BraTS2018 dataset demonstrate that DSAM achieves a 22-fold speedup compared to MedSAM and a 3.6-fold speedup over SAM-Med3D, with minor performance improvements. Consequently, DSAM facilitates SAM-based 3D medical image segmentation on standard GPU hardware.