Distilling Adversarial Robustness Using Heterogeneous Teachers

Achieving resiliency against adversarial attacks is necessary prior to deploying neural network classifiers in domains where mis-classification incurs substantial costs, e.g. self-driving cars or medical imaging. Recent work has demonstrated that robustness can be transferred from an adversarially trained teacher to a student model using knowledge distillation. However, current methods perform distillation using a single adversarial and vanilla teacher and consider homogeneous architectures (i.e., CNNs) that are susceptible to misclassify examples from similar adversarial subspaces. In this work, we develop a defense framework against adversarial attacks by distilling adversarial robustness using heterogeneous teachers (DARHT), where teachers are different with respect to architecture or adversarial training algorithm. In DARHT, the student model explicitly represents teacher logits in a student-teacher feature map and leverages multiple teachers that exhibit low adversarial example transferability (i.e., robust to different adversarial subspaces). Experiments on classification tasks in both white-box and black-box scenarios demonstrate that DARHT exhibits state-of-the-art performance; in particular, DARHT achieved at least 3.26%, 3.66%, and 25.75% improvements in weighted robust accuracy for Auto-PGD and 5.23%, 4.76%, and 22.83% for Square Attack when compared to competing adversarial distillation methods in the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets, respectively. Comparisons between homogeneous and heterogeneous teacher sets suggest that leveraging teachers with low adversarial example transferability increases student model robustness.