Transformer-Based Knowledge Distillation for Echocardiography Image Segmentation

Cardiovascular disease (CVD) is one of the major global health concerns. Real-time and automated diagnostic tools are essential due to their sudden onset and the requirement for immediate intervention. Recent advancements in neural networks, particularly Transformer-based segmentation models, have demonstrated high accuracy in diagnosing CVD. However, these models often face challenges such as large model sizes and slow inference speeds, limiting their practical use in urgent clinical settings. To address these limitations, we propose a lightweight Transformer-based model for real-time cardiac diagnostics, leveraging knowledge distillation techniques. The proposed method consists of a two-stage process where a student network learns to mimic the attention features of a pretrained teacher network. Our results demonstrate significant improvements in both Dice and Mean Intersection over Union (mIoU) scores, increasing by 0.048 and 0.071, respectively, while also achieving a reduction in model size and inference time.