Multi-objective game for fighting against Distributed Reflection DoS attacks in software-defined network

Distributed Reflection Denial of Service (DrDoS) attack represents one of the most significant threats to network security. This cyber-attack exploits vulnerabilities in existing protocols by using a botnet to send forged query packets to more than one device which are used as reflectors. As a result, a stream of replies is sent to a victim node or subnet which overwhelms it. Several security measures have been proposed to counter such attacks, unfortunately, most of them do not consider the attacker’s dynamics. Furthermore, limiting the growth of the botnet could significantly reduce the impact of such an attack. In this paper, we leverage the advantages of software-defined networks (SDN) to propose a game-theoretic approach that predicts the defender’s best moves based on Nash strategies to mitigate this attack while avoiding botnet expansion. This approach is a non-cooperative multi-objective game between the attacker which aims to (1) compromise more nodes to scale the volume of its attack, (2) launch a volumetric-based DrDoS in the network, and the defender which aims to avoid it. This game results in a mixed-strategy Pareto-Nash equilibrium. It includes a player utility-based algorithm to detect malicious flows (or nodes) and drop them (or patch them). The results of the Matlab simulation show that the proposed model is an effective means of mitigating DrDoS attacks. To the best of our knowledge, this study is the first attempt to design a defense system based on multi-objective game to counter the effects of DrDoS in SDN.