Reinforcement learning-guided segment anything model for MRI prostate and dominant intraprostatic lesions auto-segmentation

<i>Objective</i>. Accurate segmentation of the prostate and dominant intraprostatic lesions (DILs) on magnetic resonance imaging (MRI) is important for prostate cancer radiation therapy treatment planning and targeted dose escalation. However, DIL segmentation remains challenging due to small datasets, institutional bias, and variable imaging protocols. Although the segment anything model (SAM) has shown promise in medical image segmentation, most prior work depends on manual prompts. This study developed a fully automated pipeline that combines localization with a fine-tuned SAM model to segment the prostate and DIL.<i>Approach</i>. Two datasets were utilized: the PI-CAI dataset, comprising 1476 patients, and the cancer imaging archive dataset, comprising 803 patients. The pipeline consisted of two stages: (1) a reinforcement learning-based localization network predicted bounding boxes as segmentation inputs, and (2) a fine-tuned SAM model performed segmentation. Model performance was evaluated using the dice similarity coefficient (DSC), intersection over union (IoU), and detection rates, with additional analysis based on lesion volumes.<i>Main results</i>. The proposed method achieved a mean and median DSC of 0.896 ± 0.070 and 0.915, and an IoU of 0.818 ± 0.100 and 0.844 for prostate segmentation. For DIL segmentation, the mean and median DSC were 0.592 ± 0.192 and 0.636, IoU of 0.446 ± 0.190 and 0.466, with a detection rate of 89%. Four DIL groups were created based on lesion volume percentile. The mean/median DSC and IoU for each volume group are as follows: 0.5-1.0 cubic centimeters (cc): 0.555 ± 0.201/0.562 & 0.414 ± 0.205/0.391; 1.0-1.8 cc: 0.603 ± 0.185/0.660 & 0.454 ± 0.180/0.492; 1.8-4.0 cc: 0.588 ± 0.183/0.627 & 0.439 ± 0.174/0.456; >4.0 cc: 0.621 ± 0.197/0.669 & 0.477 ± 0.197/0.503.<i>Significance</i>. This study presented a fully automated prostate and DIL segmentation framework on MRI by integrating a localization network with fine-tuned SAM. The method achieved robust performance across large multi-institutional datasets and diverse lesion shapes. It shows strong potential for application to clinical workflows for prostate cancer radiation therapy planning and treatment.