SABA: Scene-aware bidirectional backdoor attack against multimodal learning

DNN models have been widely applied to multimodal tasks, including cross-modal retrieval, image captioning, and visual question answering (VQA). While unimodal DNN models face serious security threats from backdoor attacks, backdoor attacks against multimodal DNN models are still underexplored. Current multimodal backdoor attack methods mainly inherit the ideas of unimodal backdoor attacks, making them difficult to adapt to complex multimodal tasks, which results in limited generalization and weak stealthiness of backdoor triggers. We propose a scene-aware multimodal bidirectional backdoor attack method (SABA). For different benign samples, this method first assesses the scene and selects adapted trigger samples based on the semantic understanding of the scene content. It constructs a scene-aware dynamic trigger generation mechanism, generating semantically constrained image and text triggers for different types of scenes, and inserts them into benign image or text samples to achieve a backdoor that is activated in one modality and effective in another modality. We evaluated the effectiveness of this method on two tasks: cross-modal retrieval and visual question answering (VQA). Compared to various SOTA methods for multimodal and unimodal backdoor attacks, SABA demonstrates strong effectiveness. Additionally, SABA can robustly evade existing backdoor defense strategies, posing a potential threat. Furthermore, we evaluate the stealthiness of the bidirectional backdoor attack method, and the comprehensive results are superior to those of three SOTA multimodal backdoor attack methods. • Proposed a multimodal bidirectional backdoor attack. • A dynamic trigger mechanism adapting to different scenes. • Scene-aware triggers enhance attack’s adaptability and stealthiness. • Achieved high stealthiness via constrained triggers and sampling strategies.