Enhancing Adversarial Robustness of Foundation Models Without Data Centralization

This paper investigates the adversarial defense for foundation models without data centralization. To deploy foundation models in safety-critical applications, they must be robust against diverse attacks. In the defense against a single attack, foundation models require numerous adversarial examples, leading to a substantial increase in data and computational demands when addressing multiple attacks. However, traditional adversarial defense must centralize all types of adversarial examples, and there has been insufficient investigation into adversarial defense for foundation models without data centralization. Accordingly, we focus on model merging, which integrates fine-tuned models for a specific task to construct a merged model adaptable to any task. Through model merging, we aim to integrate individual models’ defensive capabilities to enhance robustness against various attacks. The main contribution of this paper is demonstrating that defense methods leveraging model merging can achieve classification accuracy approaching that of the defense method with data centralization under multiple attacks.