Recognizing Epithelial Cells in Prostatic Glands Using Deep Learning

Artificial intelligence (AI) is becoming an integral part of pathological assessment and diagnostic procedures in modern pathology. As most prostate cancers (PCa) arise from glandular epithelial tissue, an AI-based methodology has been developed to recognize glandular epithelial nuclei in prostate biopsy tissue. An integrated machine-learning network, named GlandNet, was developed to correctly recognize the epithelial cells within prostate glands using cell-centric patches selected from the core biopsy specimens. Feulgen-Thionin (a DNA stoichiometric label) was used to stain biopsy sections (4-7 µm in thickness) from 82 active surveillance patients diagnosed with PCa. Images of these sections were human-annotated, and the resultant dataset consisted of 1,264,772 segmented, cell-centric nuclei patches, of which 449,879 were centered on epithelial gland nuclei from 110 needle biopsies (training set: <i>n</i> = 66; validation set: <i>n</i> = 22; and test set: <i>n</i> = 22). The training of GlandNet used semi-supervised machine-learning knowledge of the training and validation cohorts and integrated both human and AI predictions to enhance its performance on the test cohort. The performance was evaluated against a consensus deliberation from three observers. The GlandNet demonstrated an average accuracy, sensitivity, specificity, and F1-score of 94.1%, 95.7%, 87.8%, and 95.2%, respectively, when tested on the 20,735 glandular cells found in the three needle biopsies with the visually best consensus predictions. Conversely, the average accuracy, sensitivity, specificity, and F1-score were 90.9%, 86.4%, 94.0%, and 89.7% when assessed on 57,217 cells found in the three needle biopsies with the visually worst consensus predictions. GlandNet is a first-generation AI with an excellent ability to differentiate between epithelial and stromal nuclei in core biopsies from patients with early prostate cancer.