ASGA: Attention-Based Sparse Global Attack to Video Action Recognition

Recent years have witnessed significant progress in adversarial attacks and defenses for video content understanding. Most methods craft perturbations on an independent sample basis without considering inter-frame consistency and temporal context difference. When the target model changes the frame sampling method, the success rate of the original video adversarial attack method will significantly decline. If attackers attack every frame individually, the computational cost will be too high, and the attack success rate will not improve substantially. This paper proposed a novel attention-based sparse universal attack that enhances perturbation inter-frame transferability on videos. Firstly, the key frame sequence is selected according to the feature similarity between frames. The attention map of the target model is then calculated on keyframes to identify semantically critical regions for perturbation generation. Temporal translation is further applied to improve inter-frame perturbation transferability. The integral frame perturbations are finally replicated to non-keyframes. The essential advantage of our proposed approach is that the adversarial perturbation only needs to be computed once on a keyframe sample yet can achieve universal attacks on the entire video sequence. Our proposed attack method incurs the exact computational cost of crafting perturbations for individual samples. Moreover, integrating attention maps encourages sparsity in the optimized adversarial perturbation. This paper evaluates global attack capability under two random sampling strategies: 1) randomly sampling <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> frames from all frames, and 2) randomly picking a starting frame and selecting <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N−</i>1 consecutive frames after it. Experiments on multiple models and datasets demonstrate the transferability of our approach under various frame sampling methods.