NAD-CD: Addressing concept drift in network IDS alert classification for Security Operations Centers

A Network Intrusion Detection System (network IDS or NIDS) is a widely used technology to enhance an organization’s security posture. A NIDS monitors the organizational network for cyber attacks, malware activity, and other malicious network traffic, raising alerts upon detecting such traffic. Unfortunately, NIDS technologies are known to generate large numbers of alerts daily, which can overwhelm security analysts in Security Operations Centers (SOCs). To address this, Machine Learning (ML)-based NIDS alert classification methods have been proposed in the research literature. However, previous works have not addressed the issue of concept drift in NIDS alert data. In addition, most previous works have not shared the NIDS alert datasets, and publicly available datasets are not well suited for concept drift-related research. This paper addresses these shortcomings by introducing the NIDS Alert Data with Concept Drift (NAD-CD) dataset, which spans 1.5 years and focuses on the concept drift issue. Furthermore, the paper examines passive supervised and semi-supervised ML strategies to mitigate concept drift. Finally, it demonstrates that active learning-based semi-supervised ML is an efficient solution for handling concept drift, significantly reducing the labeling workload of security analysts in SOC environments. For the sake of reproducibility, we make the NAD-CD dataset and experimental code publicly available at https://github.com/ristov/nad-cd .