Enhanced Nucleus Segmentation with Sobel Edge Detection and Attention Gate with Modified UNET

Automated segmentation of nuclei in Hematoxylin and Eosin (H&E) stained histopathology images plays a vital role in accurate cancer diagnosis and prognosis. These techniques enable detailed analysis of numerous nuclei in H&E images, providing both qualitative and quantitative insights. However, challenges arise when segmenting nuclei of varying sizes and with indistinct boundaries, which can undermine the reliability of segmentation outcomes. To address these issues, we propose a novel approach that integrates edge information, extracted from the input data, into the UNET architecture a well-established model for image segmentation. Our approach involves modifying the Attention Gate (AG) mechanism within the UNET to emphasize edge features during segmentation. This modification improves the precision of nucleus boundary delineation, particularly in cases with vague or overlapping boundaries, reducing segmentation errors and boosting overall accuracy.