AFA-MalGAN: adaptive adversarial sample generation for high-dimensional malware features via constrained manifold optimization

Abstract Deep learning-based malware detectors are vulnerable to carefully crafted adversarial examples in high-dimensional feature spaces. To bridge the gap between statistical evasion (feature-space deception) and physical validity (executable file constraints), we introduce AFA-MalGAN (Adaptive Feature-Aware Malware GAN), an adaptive adversarial generation framework grounded in constrained manifold optimization. Unlike prior GAN-based attacks that treat feature perturbation as unconstrained numerical optimization, AFA-MalGAN restricts adversarial search to a physically valid feature manifold and formulates malware adversarial generation as a structured constrained min–max game. First, we develop an Intelligent Feature Saliency Mask (ISM) that identifies decision-sensitive subspaces through momentum-smoothed gradient feedback, thereby mitigating the curse of dimensionality. Building on this mechanism, we design a Dynamic Annealing Budget (DAB) strategy that balances exploration and exploitation through time-varying perturbation boundaries in non-convex optimization. Finally, we construct a manifold-consistent multi-objective loss that enforces physical constraints of PE files, helping generated adversarial samples remain executable in real-world environments. Experimental results on the BODMAS dataset (57,293 malware samples) show that AFA-MalGAN achieves a 95.6% evasion rate against LightGBM, an industrial-grade detector, while maintaining a Feature Consistency Score (FCS) of 0.985. This work provides a constrained adversarial-generation benchmark for evaluating the robustness limits of static malware detectors under high-dimensional attacks.