A Comparative Analysis of Loss Functions in Segmentation of Medical Images with Highly Imbalanced Class Distribution: An Experimental Study for Deep Nuclei Segmentation

As is widely known, anatomical structures on medical images can be segmented successfully with the deep learning-based approaches. In such tasks, the performances of the deep learning models are also related to the loss functions, i.e. success of the network optimization in learning from the data. In this paper, a study on the comparative performance analysis of loss functions in segmentation of medical image data with unbalanced class label distribution is presented. In this context, an experimental study for deep nuclei segmentation is performed and multiple types of nuclei in colon histology images are aimed to be segmented. In our study, we have considered 8 widely known loss functions as the cross-entropy loss, dice loss, focal loss, Tversky loss, focal Tversky loss, log-cosh dice loss, L1 loss and the mean squared error loss to analyze their effects in segmentation of medical images with unbalanced data. In the experimental studies, two different segmentation tasks, semantic and instance segmentation, were also considered and the performances of the related loss functions on a recent and known dataset, CoNIC 2022, were measured and comparatively discussed.