MASKYLO: Hybrid Deep Learning Framework for Detection and Segmentation of HE Stained Histology Image

The analysis of pathology images is a crucial step of modern healthcare that helps in diagnosing tumours, determining the grade and type of tumours, planning to treat them and performing surgeries. In this study, we present MASKYLO, a novel hybrid deep learning framework that integrates YOLOv11 for real-time nuclei detection and using YOLOv8 and Mask R-CNN for precise pixel-level segmentation. MASKYLO ensures the speed and accuracy of YOLOv11 for detecting regions of interest, while the fused segmentation networks refine mask predictions to achieve high spatial fidelity. We evaluate our approach on the NuInsSeg dataset, demonstrating superior performance compared to existing methods. Our YOLOv11 detection branch achieves a precision of 0.89, a recall of 0.85, and an mAP@50 of 0.88, whereas the segmentation branch attains an IoU of 0.82 and a Dice score of 0.85, surpassing previously reported YOLOv8 and Mask R-CNN benchmarks with a 0.96 Dice Score and an IoU on the same dataset. These results highlight MASKYLO’s ability to provide accurate, efficient, and reliable detection and segmentation of histopathological images, making it a promising tool for automated pathology workflows.