Adversarial Self-Play for Adaptive Cyber Deception Against Reinforcement Learning-Based Bots in SDN-Enabled Edge Networks

Abstract Static defenses such as firewalls, signature-based detection, and fixed-rulehoneypots assume an adversary whose techniques are known in advance. Thatassumption no longer holds: bots trained by reinforcement learning probe, fail,adapt, and retry, so static countermeasures are eventually learned andbypassed. This article proposes a deception framework in which the defenderalso learns. A defender agent trained with proximal policy optimizationobserves the network through a software-defined controller and orchestrates,in real time, honeypot deployment, moving target defense mutations, and flowisolation. The confrontation is formalized as a two-agent stochastic gamedecomposed into two coupled Markov decision processes, and both policies aretrained symmetrically by adversarial self-play against a pool of pastopponents, so neither agent ever faces a frozen target. Because decoys andmutations consume bandwidth and energy, the reward explicitly charges the costof each defensive action, keeping the learned policies deployable onconstrained edge nodes. The framework is implemented on an emulated SDN/edgeinfrastructure built with Mininet and Ryu, then evaluated on two attackscenarios targeting the sensor subnetwork and the edge servers, respectively.Against a no-defense baseline and a static honeypot baseline, dynamic defenseraises the defensive success rate to about 98% on the IoT subnetwork and 97%on the edge servers, extends the time to compromise from about 10 to 50seconds in the first scenario and to 65 to 80 seconds in the second, anddrives the attacker reward to near zero while keeping reconfiguration latencybelow 50ms.