Benchmarking Adversarial Patch Selection and Location

Adversarial patch attacks threaten the reliability of modern vision models. We present PatchMap, the first spatially exhaustive benchmark of patch placement, built by evaluating over 1.5×108 forward passes on ImageNet validation images. PatchMap reveals systematic “hot-spots” where small patches (as little as 2% of the image) induce confident misclassifications and large drops in model confidence. To demonstrate its utility, we propose a simple segmentation-guided placement heuristic that leverages off-the-shelf masks to identify vulnerable regions without any gradient queries. Across five architectures-including adversarially trained ResNet-50-our method boosts attack success rates by 8–13 percentage points compared to random or fixed placements.