Do Adversarial Patches Generalize? Attack Transferability Study Across Real-Time Segmentation Models in Autonomous Vehicles

Adversarial attacks threaten deep learning models, especially in safety-critical domains like healthcare and autonomous driving. Patch-based attacks have proven effective in real-time settings due to their “drag-and-drop” nature. Building on this, we propose a novel Expectation Over Trans-formation (EOT)-based adversarial patch attack tailored for semantic segmentation (SS) in autonomous vehicles, along with a simplified, easy-to-implement loss function. We use this framework to study the cross-model transferability of patches optimized on specific SS models, including state-of-the-art CNNs (PIDNet-S/M/L) and the Vision Transformer-based SegFormer. All evaluations are conducted on the Cityscapes dataset. Our findings show that while patches generalize well across unseen images and resolutions, they transfer poorly across models-true for both CNNs and ViTs. Additionally, CNN-based models exhibit localized attack effects, unlike ViTs which display global disruptions. Per-class analysis further reveals that simple classes like ‘sky’ are more resilient to misclassification. The code for the project is available at: https://github.com/p-shekhar/adversarial-patch-transferability