An Ensemble Learning Based Nuclei Segmentation: Using Base Deep Learning Models with Different Loss Functions

Nuclei instance segmentation is a crucial task for computational pathology, as it enables precise analysis of nuclear morphology, aiding in the early diagnosis and accurate assessment of cancerous tissues. However, achieving high performance in nuclei instance segmentation is challenging due to the small size of target objects (nuclei) and their proximity to each other. Additionally, variability in staining and scanners leads to inconsistent model performance on different datasets. In this respect, the performance of stand-alone models may be inadequate for the nuclei segmentation. In this study, we introduce an ensemble model approach for nuclei instance segmentation in histopathology images, leveraging diverse loss functions to enhance segmentation accuracy. We basically employ the Kappa scores to measure the decision diversity among the base learners, ensuring the selection of models with minimal overlap in their predictions. Leveraging the proposed methodology, we identify the optimal combination of base learners on the validation dataset and subsequently apply this ensemble to the test dataset. On the CoNIC dataset, we achieve a mean panoptic quality plus of $\mathbf{5 2 . 1 2 \%}$ on our train-test split, demonstrating the potential and promise of our approach.