Transferable adversarial attacks on vision transformers via Patches Restructuring

Vision Transformers (ViTs) have achieved strong performance in image recognition, but their vulnerability to transferable adversarial examples remains an important security concern. Existing transfer-based attacks are mostly designed from the perspective of convolutional neural networks or input-level transformations, and they may not fully exploit the patch structure of ViTs when ViTs are used as surrogate models. In this paper, we propose Patches Restructuring (PR), a simple plug-in strategy for improving the transferability of adversarial examples generated from ViT surrogate models. During adversarial example generation, PR randomly shuffles the order of embedded patches with a certain probability, while keeping the model parameters and input pixels unchanged. By introducing stochastic patch arrangements into the surrogate forward pass, PR reduces the dependence of adversarial optimization on a fixed patch order and introduces greater diversity into the surrogate forward process. Extensive experiments are conducted on ImageNet classification models, including ViT-based models, CNN-based models, MLP-based models, and defended models. The results show that PR improves the black-box transferability of several representative transfer attacks under the evaluated digital-domain classification setting. These results suggest that patch shuffling inside ViT surrogate models is a useful and lightweight direction for enhancing transferable adversarial attacks.