A benchmarking framework and its validation on all evasion attacks on image classifiers listed in the adversarial robustness toolbox

What is an efficient attack against image classifiers? First, this paper defines a generic benchmarking framework to evaluate the effectiveness of evasion attacks against image classification models with relevant key performance indicators. The framework assesses attacks on 12 classifiers, for 4 datasets, and all supported attack typologies and scenarios, and specifies the tools to compare different attacks. Second, the framework is experimentally validated. We benchmark each of the 27 evasion attacks implemented in the Adversarial Robustness Toolbox, what leads to useful observations. With experiments that encompass 669,034 attack attempts and represent ∼ 7.6 GPU years of sequential computing time, our survey is the most comprehensive evaluation of evasion attacks to date. This work enables the intrinsic evaluation and comparative positioning of any new evasion attack.