Derivation of Information-Theoretically Optimal Adversarial Attacks with Applications to Robust Machine Learning

We address the theoretical problem of designing an optimal adversarial attack on a decision system to maximize degradation in system performance, as measured by the mutual information between the degraded signal and the target label. Motivated by adversarial examples in machine learning classifiers, we use an information-theoretic approach to establish conditions where adversarial vulnerability is inevitable. We derive optimal adversarial attacks for both discrete and continuous signals and demonstrate that minimizing mutual information becomes significantly harder with multiple redundant copies of the input signal. This finding supports the “feature compression” hypothesis as a basis for the adversar-ial vulnerability of deep learning classifiers. We also present computational results validating our theoretical findings.