Streaming-Based Intrusion Detection with Big Data and Online Learning Algorithms

Effective intrusion detection systems are essential for protecting computer networks amid growing cyberattacks. Traditional batch training approaches for intrusion detection in real-time streaming systems often fall short in adapting to rapidly changing attack patterns. This paper proposes two innovative methods for intrusion detection in streaming data environments: (1) a Kafka-based big data analytics framework integrating distributed streaming with deep learning for high-throughput adaptability and accuracy, and (2) an online incremental learning approach using the Hoeffding Very Fast Decision Tree classifier (VFDT) to enable responsiveness through real-time model updates. Evaluated on real-world datasets, the Kafka method achieves 98.1% accuracy and scalability under fluctuating data loads, while the Hoeffding VFDT classifier attains a 98.45% F1-score with fully adaptive incremental learning, outperforming batch methods in both responsiveness and adaptability. Threats to validity and experimental analysis demonstrate that the synergy of these methods addresses critical non-functional requirements (adaptability, accuracy, responsiveness) for dynamic intrusion detection. Future work directions include broader deployment scenarios and latency optimization.