Light-weight Vision Transformer-based Semantic Segmentation for Medical Images

Semantic segmentation is essential in medical image analysis. In this paper, we propose a lightweight vision transformer-based semantic segmentation method for medical images. The proposed method is composed of three parts: encoder, decoder, and segmentation head. The encoder is a light-weight ViT called MPiFormer developed based on iFormer and the split-and-merge strategy. A decoder is designed to fuse the multi-scale feature maps and a segmentation head is developed to predict the semantic segmentation map. Experiments on Barrett’s esophagus and Melanoma datasets demonstrate that the proposed method achieves an F1-score of 96.01% and an mIoU of 92.24% with limited annotations. Additionally, the split-and-merge mechanism reduces parameters and computation by 72.63% and 71.75%, respectively, while maintaining stable performance.