Enhancing Medical Image Segmentation with Attention-Based Knowledge Distillation

Medical image segmentation plays a critical role in healthcare, enabling accurate analysis and diagnosis through precise identification of tissues and lesions. However, existing U-Net-based segmentation models face significant challenges, including issues with data generalization and high computational resource consumption. To address these challenges, we propose a novel framework that integrates knowledge distillation and attention mechanisms to enhance both the generalization ability and computational efficiency of medical image segmentation models. We employ a teacher-student architecture for knowledge transfer, where the teacher model generates soft labels, feature embeddings, and attention maps, which are then used to guide the student model’s learning process. Additionally, attention mechanisms are incorporated to ensure that the student model focuses on the most relevant features, reducing redundant computations. We design a combined loss function that balances task-specific loss, distillation loss, and attention loss. The proposed framework is validated through extensive experiments on medical image datasets, demonstrating significant improvements in both segmentation accuracy and computational efficiency. Our results show that the student model achieves comparable performance to the teacher model while reducing training time and memory usage by over 60% on average.