NAHFormer: Neighborhood self-attention and hierarchical feature fusion transformer for image medical segmentation

Colonoscopy and dermoscopy are essential clinical tools for the early detection and diagnosis of colorectal polyps and skin lesions, respectively. Accurate segmentation of polyp and skin lesion images is critical for subsequent clinical diagnosis and treatment planning. However, the significant variations in lesion morphology, size, and boundary characteristics, together with the insufficient multi-scale feature fusion in existing methods, still pose considerable challenges to accurate medical image segmentation. To address these challenges, we propose NAHFormer, a medical image segmentation framework equipped with boundary-aware and redundancy-aware capabilities. Specifically, NAHFormer employs a pyramid-structured Mix Transformer (MiT) encoder to capture multi-scale features, enhancing the model’s generalization capability across diverse lesion appearances. A Cross-Resolution Semantic Perception (CRSP) module is designed to integrate semantic information across multiple resolutions, enabling precise delineation of lesion contours and boundaries through neighborhood attention. Furthermore, a Hierarchical Feature Fusion (HFF) module progressively aggregates multi-scale features while suppressing redundant information, thereby improving segmentation accuracy. Extensive experiments on five publicly available colonoscopy polyp datasets (Kvasir, CVC-ClinicDB, CVC-ColonDB, EndoScene, and ETIS) and two dermoscopy datasets (ISIC 2017 and ISIC 2018) demonstrate that NAHFormer consistently outperforms state-of-the-art methods, achieving mean Dice scores of 0.821 on the challenging ETIS dataset and 0.909 on ISIC 2018.