Adaptive Online Learning Approaches for Detecting Unknown Attacks in Network Traffic: A Systematic Literature Review

ABSTRACT The increasing complexity and frequency of cyberattacks present significant challenges to Intrusion Detection Systems (IDS), mainly unknown (zero‐day) attacks. IDS are essential for safeguarding networks, but traditional systems struggle to detect unknown threats in dynamic and evolving network environments due to their reliance on predefined attack signatures. Online Machine Learning models have emerged as a promising solution to address this limitation, offering adaptability and continuous learning capabilities. This Systematic Literature Review (SLR) examines the application of these models for detecting unknown attacks in network traffic. Following PRISMA guidelines and the snowballing technique, we analyze research published between 2020 and 2024, focusing on online machine learning models, adaptation mechanisms, commonly used datasets, evaluation metrics, and simulation strategies for zero‐day attacks. Our review highlights the importance of continuous model updates, diverse datasets, and integrating hybrid ML techniques to enhance detection accuracy and adaptability. Key findings include the prevalence of Active Learning, clustering models, and Autoencoders in recent research, as well as the need for more robust evaluation metrics and real‐world testing scenarios. By adopting a combined approach, this study highlights the urgency of solutions that keep pace with evolving cyber threats and establishes a new perspective that distinguishes this research from previous reviews. This innovative contribution to advancing more adaptive and effective cybersecurity solutions provides insights into state‐of‐the‐art techniques for detecting unknown attacks in dynamic network environments and outlines future research directions.