Improving Robustness and Natural Accuracy in Adversarial Distillation with Important Sample Features

Although adversarial training proves to be an effective method for training robust neural networks, its application is constrained by model capacity, making it challenging to implement in scenarios with limited computing and storage resources. To tackle this issue, adversarial distillation emerges as a viable solution to transfer knowledge from a robust teacher model to a smaller-capacity student model. However, most existing research overlooks the importance of inconsistencies in adversarial example features during distillation, which can potentially lead to a degradation in the student model's performance. In this paper, we propose a novel method named importance-aware intermediate distillation method (IID), which enhances the efficiency of distillation by leveraging the teacher model's prior knowledge to perceive the varying importance of features in different samples. Through extensive experiments on different datasets, we demonstrate the effectiveness of IID in improving model robustness.