Detecting malicious nodes using game theory and reinforcement learning in software-defined networks

Abstract Mafia, or Werewolf, is a strategic game where two teams compete to eliminate each other’s players through deception and hidden roles. The game dynamics and role interactions share notable similarities with adversarial behaviors in network security, making it a valuable framework for modeling cyber threats, particularly botnet detection. In this paper, we introduce a novel game-theoretic approach to botnet detection, leveraging the strategic deception dynamics of the Mafia game to model adversarial behavior in cybersecurity. We present a mathematical model for Mafia games, formulating winning strategies for different roles using linear relations and reinforcement learning techniques. Furthermore, we establish a direct mapping between Mafia game roles and network security components, illustrating how botnet attack patterns align with hidden-role game mechanics. Our proposed detection strategies are applied to real-world network attack scenarios, demonstrating their effectiveness in mitigating botnet threats. We evaluate the model using applicable security metrics and compare the results with existing detection methodologies to validate the approach. Our findings indicate that the suggested strategies improve detection accuracy by 12% over conventional methods. Additionally, we conduct network emulations using Mininet, simulating Mirai botnet infections. The results show that the true positive and true negative detection rates for a network modeled by the Mafia game framework reach 71% and 91%, respectively. These insights provide a foundation for integrating deception-based modeling into modern intrusion detection systems, enhancing network resilience against adaptive cyber threats.