Trade-Offs in Deep Learning Model Loss and Configuration for Sparse Histological Segmentation: A Case Study in Pediatric Ileal Histology

Tissue segmentation is an important task for histological deep learning. However, when a tissue is rare, successful segmentation is more difficult and sensitive to the choice of loss function. This work systematically evaluates the impact of loss function choice over precision, sensitivity, Jaccard index, and Dice coefficient, with multiple model configurations to clarify the trade-offs in this choice. Experiments are conducted for muscularis tissue segmentation in pediatric ileal histological images at the specimen level. Surprisingly, Dice loss alone did not produce models with stronger precision than sensitivity. Rather the converse was true, models trained using Dice loss tend toward over-prediction of the target whereas models trained using Dice loss of the negative masks (absence of the target) are more conservative when predicting the positive class, producing higher precision and lower sensitivity. The segmentation model that achieved the highest AUROC score was a U-Net based model with multiple decoders trained using weighted negative combined Dice loss.