Improving robustness of CLIP by adversarial training enhanced by brain activity

Contrastive Language-Image Pre-training (CLIP) is vulnerable to adversarial attacks which cause misclassification by subtle modifications undetectable to the human eye. Although adversarial training strengthens CLIP models against such attacks, it often degrades their accuracy on clean images. To tackle this challenge, we propose a novel defense strategy that leverages human brain activity data. The proposed method combines features of brain activity with those of adversarial examples, which enhances the robustness of CLIP while maintaining high accuracy on clean images. Experimental results demonstrate the effectiveness of our method for the accurate retrieval of clean and adversarial images. These results highlight the potential ability of brain data to overcome the existing challenges of adversarial defense in foundation models.