Convex Shape Prior for Deep Convolution Neural Network-Based Image Segmentation

Abstract Convex shapes (CSs) are common priors for image segmentation. It is important to design proper techniques to represent CS. So far, it remains a challenge to guarantee that the output objects from deep convolution neural networks (DCNNs) are CS. In this work, we propose a convex shape technique that can be easily integrated into the commonly used DCNN architectures and guarantee that outputs are CS. This method is flexible, and it can handle multiple objects and allow some of them to be convex. Our method is based on the dual representation of the sigmoid activation function in DCNNs. Moreover, our method can integrate spatial regularization and other shape priors by using a soft threshold dynamics (STD) method. This regularization can make the boundary curves of the segmented objects simultaneously smooth and convex. We design a very stable active set projection algorithm to solve our model numerically. This algorithm can form a new plug-and-play DCNN layer called CS-STD, whose outputs must be a nearly binary segmentation of convex objects. In the CS-STD block, the convexity information can be propagated to guide the DCNN in both forward and backward propagation during the training and prediction. As an application example, we apply the convexity prior layer to the segmentation of the retinal fundus image using the popular U-Net, BCDU, MedSAM and DeepLabV3+ as the backbone networks. Experimental results on several public datasets show that our method is efficient and outperforms classic DCNN segmentation methods.