Multi-modal segment anything model (mmSAM) for tumor segmentation in multi-tracer oncologic PET/CT

Lesion segmentation is important in targeted radionuclide therapy dosimetry. In this study, we proposed a novel 2D multi-modal Segment Anything Model (mmSAM) for lesion segmentation in whole-body multi-tracer PET/CT images. The AutoPET 2024 dataset, including 18F-PSMA (369 subjects), 18F-FDG (170 subjects) and 68Ga-PSMA (170 subjects) PET/CT images and their corresponding tumor masks, was used in this study. The 18F-PSMA PET/CT dataset was used as the primary dataset and was divided into 233: 36: 100 for training, validation and testing the mmSAM, using both PET and CT input. The transferability of the primary model was evaluated on 100 patients on other 2 datasets, without and with fine-tuning using 70 cases of 18F-FDG and 68Ga-PSMA respectively. Standard single modal 2D SAM with only PET input, 3D nnUNet with two-channel PET/CT input and the thresholding-based method were also implemented for comparison. Mean Dice, the 95th percentile Hausdorff distance (mean HD95), mean standardized uptake value (mean |SUVmean| error), metabolic tumor volume (mean |MTV| error), true positive rate (TPR), positive predictive value (PPV), and false discovery rate (FDR) were computed for all tumors. Statistical significance among different segmentation methods was evaluated using the Wilcoxon test. mmSAM achieved the best performance as compared to other segmentation methods for the primary 18F-PSMA dataset (mean Dice/HD95/|SUVmean| error/|MTV| error/TPR/PPV/FDR = 0.76/1.92 mm/5.10%/14.60%/100%/96.51%/3.49%, all p < 0.05), without fine-tuning (mean Dice/HD95/|SUVmean| error/|MTV| error/TPR/PPV/FDR = 0.61/2.28 mm/ 15.83%/27.71%/100%/78.91%/21.09% for 18F-FDG; 0.77/1.33 mm/5.55%/17.96%/100% /97.78%/2.22% for 68Ga-PSMA, p < 0.05), as well as with fine-tuning (mean Dice/HD95/|SUVmean| error/|MTV| error/TPR/PPV/FDR = 0.65/1.81 mm/6.62%/13.93%/100%/ 80.50%/19.50% for 18F-FDG; 0.81/1.15 mm/4.62%/14.30%/100%/99.30%/0.70% for 68Ga-PSMA, p < 0.05) on the cross-tracer datasets. The proposed mmSAM is promising for lesion segmentation in multi-tracer oncologic PET/CT images. Fine-tuning significantly enhances segmentation accuracy for cross-tracer studies.