ETiSeg-Net: edge-aware self attention to enhance tissue segmentation in histopathological images

Abstract Digital pathology employing Whole Slide Images (WSIs) plays a pivotal role in cancer detection. Nevertheless, the manual examination of WSIs for the identification of various tissue regions presents formidable challenges due to its labor-intensive nature and subjective interpretation. Convolutional Neural Network (CNN) based semantic segmentation algorithms have emerged as valuable tools for assisting in this task by automating ROI delineation. The incorporation of attention modules and carefully designed loss functions has shown promise in further augmenting the performance of these algorithms. However, there exists a notable gap in research regarding the utilization of attention modules specifically for tissue segmentation, thereby constraining our comprehension and application of these modules in this context. This study introduces ETiSeg-Net (Edge-aware self attention to enhance Tissue Segmentation), a CNN-based semantic segmentation model that uses a novel edge-based attention module to achieve effective delineation of class boundaries. In addition, an innovative iterative training strategy is devised to efficiently optimize the model parameters. The study also conducts a comprehensive investigation into the impact of attention modules and loss functions on the efficacy of semantic segmentation models. Qualitative and quantitative evaluations of these semantic segmentation models are conducted using publicly available datasets. The findings underscore the potential of attention modules in enhancing the accuracy and effectiveness of tissue semantic segmentation.