<scp>TCSAFormer</scp> : Efficient Vision Transformer With Token Compression and Sparse Attention for Medical Image Segmentation

ABSTRACT In recent years, transformer‐based methods have achieved remarkable progress in medical image segmentation due to their superior ability to capture long‐range dependencies. However, these methods typically suffer from two major limitations. First, their computational complexity scales quadratically with the input sequences. Second, the feed‐forward network (FFN) modules in vanilla Transformers typically rely on fully connected layers, which limits models' ability to capture local contextual information and multiscale features critical for precise semantic segmentation. To address these issues, we propose an efficient medical image segmentation network, named TCSAFormer. The proposed TCSAFormer adopts two key ideas. First, it incorporates a Compressed Attention (CA) module, which combines token compression and pixel‐level sparse attention to dynamically focus on the most relevant key‐value pairs for each query. This is achieved by pruning globally irrelevant tokens and merging redundant ones, significantly reducing computational complexity while enhancing the model's ability to capture relationships between tokens. Second, it introduces a Dual‐Branch Feed‐Forward Network (DBFFN) module as a replacement for the standard FFN to capture local contextual features and multiscale information, thereby strengthening the model's feature representation capability. We conduct extensive experiments on four publicly available medical image segmentation datasets: ISIC‐2018, CVC‐ClinicDB, Synapse and Abdomen MRI, to evaluate the segmentation performance of TCSAFormer. Experimental results demonstrate that TCSAFormer achieves superior performance compared to existing state‐of‐the‐art (SOTA) methods, while maintaining lower computational overhead, thus achieving an optimal trade‐off between efficiency and accuracy. The code is available on GitHub.