SNEAK: Synonymous Sentences-Aware Adversarial Attack on Natural Language Video Localization

Natural language video localization (NLVL) is an important task in the vision-language understanding area, which calls for an in-depth understanding of not only computer vision and natural language side alone, but more importantly the interplay between both sides. Adversarial attack has been well-recognized as a critical security issue of deep neural network models, which requires prudent investigation. Despite its extensive yet separated studies in video and language two kinds of unimodal tasks, current understanding of the adversarial attack in NLVL tasks including both the vision and language modality is less developed. This paper therefore aims to comprehensively investigate the adversarial attacks of NLVL models by examining three different facets of adversarial attack methods. To achieve the attack goal, we propose a new adversarial attack paradigm called synonymous sentences-aware adversarial attack on NLVL (SNEAK), which captures the cross-modality interplay between the vision and language sides. To further enhance the stealthiness of SNEAK, we propose a frame importance-guided pruning with SNEAK attack (PSA) mechanism to reduce the amount of perturbed frames. Extensive experiments on five NLVL models and two datasets demonstrate the effectiveness and stealthiness of the proposed attack methods.