A Dual-Teacher and Multi-Scale Adversarial Example-Based Framework for Adversarial Distillation

Deep neural networks (DNNs) are vulnerable to adversarial attacks, while conventional adversarial training often degrades accuracy on clean examples and lacks sensitivity to specific-scale perturbations. This paper introduces DTAD-MS, a novel adversarial distillation framework using dual teachers and multi-scale adversarial examples to address these limitations. DTAD-MS distinctively integrates three components: 1) multi-scale adversarial example generation and fusion, leveraging multi-scale feature extraction and deep attention to bolster robustness against varied perturbation scales; 2) a dual-teacher strategy where a clean teacher preserves clean accuracy and a robust teacher enhances adversarial robustness, balanced by an adaptive loss function; and 3) attention-guided distillation, using the clean teacher’s attention maps to direct the student model’s focus on critical features, thereby improving robust feature assimilation. Experimental results on CIFAR-10/100 demonstrate that DTAD-MS improves Weighted Robust Accuracy (W_Robust_Acc) by up to 2.0% (CIFAR-10) and 1.5% (CIFAR-100) for ResNet-18, and by 0.5% (CIFAR-10) and 3.0% (CIFAR-100) for MobileNet-V2, achieving an effective balance.