Multiscale hard mining distillation for multiorgan segmentation

Knowledge distillation is a technique that involves transferring knowledge from a teacher network to a student network, enabling the student to benefit from fewer parameters and computations while maintaining a comparable accuracy to the teacher. In this paper, we propose a novel distillation method that incorporates two key modules: Hard Mining Distillation and Multi-scale Feature Distillation. The Hard Mining Distillation module builds upon the vanilla knowledge distillation approach, taking inspiration from Venn diagrams. By introducing this module, we aim to enhance the students' learning by focusing on challenging areas. Moreover, we integrate the Hard Mining Distillation module into the multi-scale feature distillation process to enable more detailed learning. This method is straightforward to implement and lightweight, eliminating the complexities associated with previous studies while ensuring efficient distillation. To evaluate the effectiveness of our proposed method, we conducted quantitative experiments on the BTCV dataset. The experimental results demonstrate that our method outperforms other approaches, achieving the best performance in terms of accuracy.