Smart neural network and cognitive computing process for multi task nuclei detection segmentation and classification in breast cancer histopathology images

The detection, segmentation, and differentiation of benign and malignant nuclei from the histopathology images is a challenging task for the early diagnosis of breast cancer. Misinterpretation of True Negative (TN) and False Positive (FP) can generate incorrect results. The proposed Cognitive Computing Process (CCP) detects and segments the nuclei using Deep U-Net with Spatial Attention Mechanisms (SAM) and microns-per-pixel measurements to accurately locate and assess nuclei density. To separate the nuclei of benign and malignant, the patches are introduced to leverage the model's learning process. The proposed Smart Neural Network (SNN) models contain Smart Convolutional Neural Network (SCNN) and Deep Convolutional Neural Network (DCNN) to reduce incorrect results. Proposed CCP and SNN were evaluated using the BreakHis dataset, which contains 5547 images of benign and malignant samples at various magnifications (40×, 100×, 200×, 400×). These images were processed into patches, totaling 11,642, 9282, 9102, and 9678 patches, each 224 × 224 pixels. The CCP model outperformed state-of-the-art models UNet, Residual UNet (ResUNet), and Convolutional Neural Network Long Short-Term Memory (CNN-LSTM) with a Dice coefficient of 99.90%, an F1-score of 99.04%, a precision of 99.80%, and a recall of 99.76%. The learning process began with a learning rate of 0.01 and a decay rate of 0.8, and the SCNN achieved false negative and false positive rates of 0.04 and 0.05 for low-density nuclei at 400× and 40× magnification, respectively. In contrast, the Deep Convolutional Neural Network (DCNN) recorded rates of 0.02 and 0.01. For high-density patches, the SCNN model FN and FP rates of 0.0 and 0.08, while the DCNN reported 0.09 and 0.0. The proposed learning process with Smart Neural Networks (SNN) achieved high precision (77-99%), recall (75-99%), F1-score (75-99%), and an AUC of 86-100%. The combination of CCP and SNN improved accuracy over existing CNN models like ResNet50, VGG19, DenseNet109, DenseNet201, and VGG16. An ablation study showed a p-value of 0.00003 based on the AUC, highlighting the model's potential to enhance automated breast cancer diagnosis and support clinical decision-making.