Optimizing knowledge distillation for efficient breast ultrasound image segmentation: Insights and performance enhancement

Most modern models designed for ultrasound (US) image segmentation are characterized by high computational and memory requirements, limiting their practical utility in point-of-care US settings. Consequently, researchers have devised innovative approaches to compress these large models, enabling the training of smaller networks capable of achieving comparable generalization performance. Among these strategies, knowledge distillation (KD) has emerged as particularly suitable for scenarios involving small datasets or where significant efficiency improvements are desired. While previous KD-based methods have focused on extracting comprehensive information from diverse levels of teacher representation, they often overlook the identification of the most effective representation level. Additionally, many existing techniques propose intricate strategies that present implementation challenges. To address this gap, our study concentrates on selecting optimal teacher representations from various levels. Through an exhaustive analysis of KD pathways, loss functions, and the impact of augmentation, we offer valuable insights into the mechanisms underlying knowledge transfer from the teacher to the student networks. Our proposed methodology significantly enhances student performance, elevating the Dice similarity score from 73% to 80%, while the teacher model achieves 81%. Notably, our student model achieves this improvement with only 0.82 million parameters, compared to the teacher model’s 96 million parameters.