Evolving cybersecurity frontiers: A comprehensive survey on concept drift and feature dynamics aware machine and deep learning in intrusion detection systems

Intrusion Detection Systems (IDS) have become pivotal in safeguarding information systems against evolving threats. Concurrently, Concept Drift presents a significant challenge in machine learning, affecting the adaptability and accuracy of predictive models in dynamic environments. Understanding the synergy between IDS and Concept Drift is crucial for developing robust security systems. The motivation behind this survey is driven by the emerging complexities in cyber threats and the dynamic nature of data streams, which necessitate advanced IDS capable of adapting to Concept and Feature Drift. Our analysis reveals a glaring omission in the existing literature—the integration of Concept Drift and Feature Drift within IDS. Most studies have focused on Concept Drift in a general context or on IDS but have yet to comprehensively consider the implications of data dynamics. This oversight has led to a fragmented understanding and suboptimal approaches to tackling modern cyber threats. To address this, we propose a comprehensive review that delves into the role of machine learning in IDS, explicitly focusing on Concept and Feature Drift. We have proposed a framework that includes all the necessary components for a drift-aware IDS. The framework incorporates dynamic feature selection, adaptive learning algorithms, and continuous monitoring techniques to handle Concept Drift and Feature Drift effectively. The survey highlights state-of-the-art methodologies and current challenges in integrating these concepts. The methodology involves an exhaustive analysis of published works from 2019 to 2024, comparing and contrasting various models and approaches. This includes a detailed examination of Concept Drift-aware IDS methods, dynamic feature selection techniques, and the impact of high dimensionality in IDS. These quantitative improvements underscore the necessity for developing adaptive and resilient IDS. The survey uncovers under-represented areas in current research, paving the way for future investigations. By highlighting these gaps and providing comparative data, the survey sets a clear direction for upcoming research efforts to foster the development of more dynamic and adaptable IDS solutions. The quantitative experimental evaluation of the proposed framework is planned to be conducted in a future article, where we will assess its effectiveness and performance in real-world scenarios. • The survey explores ML's role in enhancing IDS accuracy and adaptability against evolving cyber threats. • Emphasizes the scarcity of research on concept and feature drift in IDS, crucial for effective cyber threat defense. • Reviews recent methodologies impacting IDS, comparing models and approaches. • Identifies underexplored research areas, urging future studies on dynamic and adaptable IDS solutions. • Proposes a novel framework for IDS challenges, advocating a multi-layered approach with dynamic feature selection and reinforcement learning.