Quality Analysis in IDS Dataset: Impact on Model Generalization

Intrusion Detection Systems (IDS) utilizing machine learning (ML) are essential for detecting network intrusions. A key challenge in ML-based IDS is ensuring models can generalize effectively—accurately classifying unseen data. This ability largely depends on the quality of training and testing datasets. This paper underscores the importance of high-quality datasets and details how errors in dataset creation can significantly impair model generalization. Through a comprehensive analysis of the CREMEv2 dataset as a case study, chosen for its potential due to reproducibility and structured attack lifecycles, we identified several key errors, including overlapping attack techniques (identical code used for different attacks), SSH traffic contamination (non-targeted SSH traffic mislabeled as malicious), benign traffic contamination (normal traffic mislabeled as malicious), and unexecuted traffic patterns (expected traffic for certain attacks not generated). These errors introduce inaccuracies, severely impacting the generalization performance. By rectifying these issues, we achieved a marked improvement in model performance, demonstrated by an increase in the average F1 score from 0.25 to 0.8. The methodologies and insights from this study can be universally applied to other datasets, helping to uncover hidden errors and enhance generalization performance.