Characterizing Natural Adversarial Examples Through Activation Map Analysis

ABSTRACT Adversarial examples are an intriguing and critical topic in the field of machine learning. The impact of malignant perturbations on deep learning‐based systems, especially in safety‐critical applications, highlights a significant security concern. While most research has focused on artificially generated adversarial attacks–crafted through optimization algorithms and constrained perturbations, it is important to note that adversarial examples can also occur naturally, without any artificial manipulation, during the prediction of real‐world images. These naturally occurring adversarial examples pose unique challenges, as they are harder to detect and interpret. Despite their importance, the study of natural adversarial examples remains in its early stages. Fundamental questions remain unanswered: Do natural adversarial examples exhibit similar behaviours or properties as artificially generated ones? How should models be adapted to improve their robustness against such natural inputs? To address these questions, this work proposes an in‐depth analysis of activation maps to compare the internal behaviour of neural networks when processing clean images, artificially perturbed inputs and natural adversarial examples. A set of quantitative metrics is extracted from activation heatmaps at various network layers, including mean activation intensity, centroid displacement and standard reference image quality metrics. These measurements enable a systematic comparison of how the network attends to different image regions under varying conditions. The experimental results demonstrate that natural adversarial examples exhibit statistically significant differences in activation patterns compared to their artificial counterparts, suggesting that they may require distinct strategies for detection and defence.