Robust Adversarial Defense Based on Non-Transferability of Attack Across Foundation Models

This paper presents a novel adversarial defense method that exploits non-transferability of attack across foundation models. Existing adversarial training methods have insufficient robustness to adversarial examples under powerful adversarial attacks in a white-box setting. We clarify that there is no attack transferability between various foundation models through preliminary experiments and propose a defense method that combines the non-transferability with adversarial training. The proposed method constructs a common embedding space of different models and detects adversarial examples by leveraging the different outputs from each model. Based on the detection of adversarial examples, the proposed method can adaptively select a robust model depending on the input images based on non-transferability of attack between models. Experimental results show that the proposed method improves the accuracy of zero-shot class classification for adversarial examples against each model.