Small-object-sensitive deep reinforcement learning for fully automatic 3D vessel segmentation in medical images

Pre-extracted lumen information of 3D vessel in medical images can effectively assist doctors in intraoperative navigation and postoperative evaluation, which has important clinical value. The main challenge faced by fully automatic 3D vessel segmentation comes from the imbalanced proportion of the vessels in medical image, which may lead to lost target. In this paper, a fully automatic 3D vessel segmentation method based on small-object-sensitive deep reinforcement learning, is presented. The region of target is firstly detected by the bounding box of a deep reinforcement learning (DRL) network, and then is segmented with a convolutional neural network (CNN). To better detect small vessel object, we have made three improvements to the existing DRL-based detection network: 1) A novel state with random receptive field expansion is applied to provide the agent with necessary information even if part of the target is lost. 2) A Recall-priority reward is presented to provide the most complete region for the next segmentation stage. 3) The dependency of vascular spatial positions between adjacent slices is utilized to correct the errors in detection stage, and the topological integrity of the obtained vascular structure is improved. The proposed method has been extensively validated on a challenging vessel dataset with 100 computed tomography angiography (CTA) scans. The segmentation accuracy of this method is Dice=93.75%, which outperforms the baseline and other automatic 3D vessel segmentation algorithms. This method has advantages in positioning accuracy, segmentation accuracy, and operational efficiency, and can be easily applied to clinical applications.