Boosting Adversarial Robustness through Channel Adversarial Distillation

Adversarial attacks have become a significant challenge for deep learning models, especially in resource-constrained environments. While adversarial training (AT) has proven effective for improving robustness, it often suffers from limited applicability to smaller models due to their computational constraints. Recently, adversarial distillation has emerged as a promising approach to transfer robustness from large teacher models to smaller student models. However, existing adversarial distillation methods often focus on logits-level distillation and overlook the importance of intermediate layer features. In this paper, we propose a novel method called Channel Adversarial Robustness Distillation (CARD) which attempts to identify the vital knowledge against adversarial threats through channel distillation. CARD leverages feature alignment at the channel level between the teacher and student models, utilizing 1x1 convolutions to ensure compatibility in feature map dimensions. We introduce Channel Distillation Loss (CD), which computes the divergence between aligned feature maps, and incorporate adaptive KL divergence for the upper bound of prediction discrepancy. The experimental results demonstrate that CARD effectively enhances adversarial robustness while maintaining high accuracy, offering a balanced approach to model distillation. Our method provides a promising direction for improving the robustness of student models in adversarial settings.