Risk‐Aware SDN Defense Framework Against Anti‐Honeypot Attacks Using Safe Reinforcement Learning

ABSTRACT The development of multiple attack methods by external attackers in recent years poses a huge challenge to the security and efficient operation of software‐defined networks (SDN), which are the core of operational controllers and data storage. Therefore, it is critical to ensure that the normal process of network interaction between SDN servers and users is protected from external attacks. In this paper, we propose a risk‐aware SDN defense framework based on safe reinforcement learning (SRL) to counter multiple attack actions. Specifically, the defender uses SRL to maximize the utility by choosing to provide a honeypot service or pseudo‐honeypot service within predefined security constraints, while the external attacker maximizes the utility by choosing an anti‐honeypot attack or masquerade attack. To describe the system risk in detail, we introduce the risk level function to model the simultaneous dynamic attack and defense processes. Simulation results demonstrate that our proposed risk‐aware scheme improves the defense utility by 17.5% and 142.4% and reduces the system risk by 42.7% and 59.6% compared to the QLearning scheme and the Random scheme, respectively.