CNN Ensembles for Nuclei Instance Segmentation in OED Histological Images

Cell nuclei segmentation in histopathological images is essential for diagnosing oral epithelial dysplasia, a condition associated with an increased risk of oral cancer. Deep learning models have demonstrated significant potential in this task, but challenges persist due to variations in staining, tissue morphology, and artifacts. This study investigates segmentation models and proposes ensemble approaches to improve instance segmentation in OED histological images. The ensemble integrates diverse segmentation models using different voting rules, with the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$D_{C^{-}}$</tex> weighted averaging achieving the best results. The proposed method obtained an accuracy of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{9 4. 0 9 \%}$</tex> and a Dice coefficient of 0.9461, surpassing individual models and demonstrating significant improvement. Comparative analysis with the literature shows that the ensemble achieved competitive performance across multiple datasets. These results reinforce the potential of ensemble learning to enhance segmentation accuracy, contributing to the development of robust computer-aided diagnosis systems.