Adversarial Attack and Detection in Token‐Pruned Large Vision‐Language Models

ABSTRACT Large vision‐language models (LVLMs) incur high computational costs, and visual token pruning is commonly used to remove redundant tokens and enhance efficiency. However, from a security perspective, such acceleration mechanisms may have a non‐monotonic impact on adversarial robustness: They may either improve robustness by removing perturbation‐related evidence or reduce robustness by discarding semantically important visual tokens. In this paper, we first analyze the setting‐dependent robustness effects of dynamically pruned LVLMs. Building on this analysis, we propose an attention‐flip attack that manipulates pruning‐related attention redistribution by suppressing high‐importance visual tokens while promoting low‐importance regions, thereby increasing the likelihood that discriminative visual evidence is discarded during inference. To counter this threat, we further propose a micro‐feature‐based adversarial detector. Rather than relying on global attention patterns, the detector captures local distortions around the pruning boundary using a multidimensional feature representation, including threshold‐neighborhood density, truncated residual energy ratio, mask spatial dispersion, and statistical descriptors. Experimental results show that, under the same perturbation budget and iterative setting, the proposed attack consistently achieves higher attack success rates (ASRs) than standard first‐order attack baselines on pruned Qwen2‐VL and LLaVA‐OneVision models. For adversarial examples generated by the proposed Attention‐Flip attack, the detector achieves accuracies of 94.79% on Qwen2‐VL and 93.58% on LLaVA‐OneVision. It further reaches 90.37% on standard PGD examples generated from an unpruned Qwen2‐VL model.