Efficient merging and validation of deep learning-based nuclei segmentations in H&E slides from multiple models

Characterizing cellular composition in tissue samples offers fundamental insights into functional and biological processes. Understanding the abundance or lack of specific cell types, such as inflammatory cells in the context of microenvironments such as tumor can help guide disease progression and personalized medicine. Several clinical laboratory methods to characterize the cellular composition are limited by scalability and high-costs. Digitizing pathology slides and applying deep learning (DL) models have enabled efficient and cost-effective nuclei segmentation and cell type quantification; however, the DL-models are limited by their inability to segment specific cell types and specific models may be more effective than others at certain tasks. Consequently, there remains a need for methods that leverage the strengths of multiple models to efficiently integrate nuclei segmentation for various cell types. In this study, we propose a novel solution for integrating nuclei segmentation from multiple DL-methods on hematoxylin and eosin slides from 471 normal prostate samples and highlight the limitations of using a single DL-method. We validate the DL-derived cell type proportions, by comparing against estimates from a manual pathologist review and show that the integrated approach results in higher concordance over the individual models. We further validate the derived cell type proportions from the DL-methods by their ability to explain the variance of RNA gene expression. The integrated approach yields robust cell type proportions that explain the variance of the gene expression with 12% and 22% relative improvement than current state-of-the-art model and manual pathologist review, respectively. The subset of 403 genes with high explained variation (>30%) by epithelial proportion were significantly enriched for relevant biological pathways. These findings indicate that ensemble approaches to nuclei segmentation and cell-type classification may provide more accurate representations of cellular composition from digitized slides.