Channel attention probability knowledge distillation for skin disease classification

Knowledge distillation shows promising results in classifying dermatology images. Now, we propose a new framework including two distillation modules. The first method based on channel attention transfers each channel information of the teacher's intermediate features to the students as knowledge, which is called channel attention distillation. The second is to allow students to learn more detailed knowledge of probability distributions in the teacher's logits, instead of just imitating the teacher's logits. Our proposed method is experimented on MonkeyPox and Dermnet datasets and compared with other knowledge distillation methods. The experimental results show that our proposed method is better than other methods, especially when the teacher-student is resnet50-resnet18, the student performance is close to the teacher model.