A Survey on Reinforcement Learning-Driven Adversarial Sample Generation for PE Malware

Malware remains a central tool in cyberattacks, and systematic research into adversarial attack techniques targeting malware is crucial in advancing detection and defense systems that can evolve over time. Although numerous review articles already exist in this area, there is still a lack of comprehensive exploration into emerging artificial intelligence technologies such as reinforcement learning from the attacker’s perspective. To address this gap, we propose a foundational reinforcement learning (RL)-based framework for adversarial malware generation and develop a systematic evaluation methodology to dissect the internal mechanisms of generative models across multiple key dimensions, including action space design, state space representation, and reward function construction. Drawing from a comprehensive review and synthesis of the existing literature, we identify several core findings. (1) The scale of the action space directly affects the model training efficiency. Meanwhile, factors such as the action diversity, operation determinism, execution order, and modification ratio indirectly influence the quality of the generated adversarial samples. (2) Comprehensive and sensitive state feature representations can compensate for the information loss caused by binary feedback from real-world detection engines, thereby enhancing both the effectiveness and stability of attacks. (3) A multi-dimensional reward signal effectively mitigates the policy fragility associated with single-metric rewards, improving the agent’s adaptability in complex environments. (4) While the current RL frameworks applied to malware generation exhibit diverse architectures, they share a common core: the modeling of discrete action spaces and continuous state spaces. In addition, this work explores future research directions in the area of adversarial malware generation and outlines the open challenges and critical issues faced by defenders in responding to such threats. Our goal is to provide both a theoretical foundation and practical guidance for building more robust and adaptive security detection mechanisms.