An Automated Benchmarking Framework for Anomaly-based Intrusion Detection Systems

The rapid evolution of cyber threats has set an urgent requirement for cyber security solutions. In response to this, anomaly-based IDSs, powered by artificial intelligence, have emerged as a promising solution for detecting novel threats. However, the development of these systems is hindered by the time-consuming data preparation process and the absence of standardized evaluation frameworks. To address these challenges, this paper introduces a comprehensive benchmark framework designed to automate the evaluation of anomaly-based IDS solutions. The framework streamlines data preparation by incorporating multiple datasets and preprocessing steps, enabling researchers to more focus on model development. Additionally, we present baseline results for integrating machine learning models into IDSs by evaluating six models on five popular datasets: CIC-IoT2023, CIC-DDoS2019, UNSWNB15, CIDDS001 and CIC-IDS2018. These results demonstrate the effectiveness of our framework and offer valuable insights for integrating machine learning models into IDS implementations.