Enhancing Adversarial Robustness in Convolutional Neural Networks by Integrating Human Viewing Strategies

Adversarial examples pose a serious threat to the reliability of deep neural networks by subtly manipulating inputs to induce incorrect outputs. Despite progress in the field, convolutional neural networks (CNNs) remain vulnerable to such perturbations. This research proposes a novel approach to improve adversarial robustness of CNNs by incorporating human viewing behavior into both training and testing processes. To establish a baseline for comparison with models trained from scratch, an eye-tracking dataset—collected during experiments involving novel objects from the Greeble dataset—was used to capture how humans learn to classify previously unseen object categories. This study compares model saliency with human fixation maps and introduces a padded patch-based training pipeline alongside gaze-informed testing modes. Our study introduces three training modes and six testing modes, all designed to reflect human viewing strategies — such as training on full images, scaled, and padded patches, and testing on corresponding variants, including gaze-informed scaled and padded patches and heatmap-overlaid inputs. This design mimics how human attention is limited to a narrow visual angle rather than the entire field of view. Experimental results reveal that participants employed significantly distinct viewing patterns while examining the same images. This suggests that no single viewing strategy is universally adopted. Nevertheless, training with inputs that simulate a narrow visual angle (padded patch-based training) proved more effective than using full images during training. Models tested on patches maintain stable accuracy even as the strength of adversarial perturbations increases. Notably, all models achieve their best performance when evaluated on heatmap-overlay images and padded patches informed by human gaze. These findings suggest that aligning model input with human visual attention can improve both performance and adversarial robustness at inference time.