Evolving botnet defenses: A survey of machine learning approaches for identifying polymorphic and evasive malware

The advancement of polymorphic and evasive malware helps botnets overcome traditional security mechanisms, rendering them obsolete. This fact, along with the sophisticated growth of botnets, poses a threat to modern computer networks. As cyber threats evolve, so must the strategies used to detect and mitigate them. This paper highlights the various machine learning (ML) techniques employed for botnet detection, outlining their advantages, limitations, and practical applications. The study analyzes supervised, unsupervised, and deep learning approaches and examines their role in detecting malicious network behavior. It is discovered that although the ML-based detection systems provide promising solutions, exposing the detection system to a real-world scenario uncovers more issues like adversarial resistance, scalability, and computational overhead. Furthermore, this paper brings attention to new issues such as providing strong defenses against adversarial attacks and the use of explainable AI for a better understanding of their purpose. With the goal of improving the state of botnet defense, this research aims to provide comprehensive methodologies while underscoring existing gaps toward ensuring continuous development in robust cybersecurity strategies driven by machine learning.