Feature Aggregation and Mixing via Cross Attention for Nuclei Instance Segmentation

Nuclei instance segmentation aims to identify individual nuclei within histopathological images, playing a crucial role in various medical image analysis tasks such as cancer diagnosis and treatment planning. However, this task encounters challenges such as densely clustered nuclei, blurry boundaries, overlapping regions, and variations in shape and size across different organs, leading to the erroneous separation of individual nuclei. In recent years, transformer architecture has achieved commendable performance in medical image segmentation due to its ability to efficiently capture global contextual information. However, its inherent design limits its ability to capture fine-grained local information within images, thereby restricting its full potential. To address this limitation, we propose a novel transformer-based method and design two effective modules, a multi-scale feature aggregation module (MSFAM) and a feature-mixing cross-attention module (FMCAM). These modules are specifically designed to aggregate feature maps by capturing fine-grained information and enhancing contextual details through mixing feature maps of various scales via cross attention. Furthermore, to separate each nucleus, we add an additional branch to predict horizontal and vertical distance maps to obtain boundary information, thereby enabling the distinction of each nucleus. We evaluate the proposed method on the multi-organ nuclei segmentation (MoNuSeg) dataset. Experimental results demonstrate that our method achieves outstanding performance in the nuclei instance segmentation task.