Unveiling the Generalizability Gap: A Cross-Domain Evaluation of Machine Learning Algorithms for Network Intrusion Detection

The ever-evolving threat landscape necessitates highly adaptable Network Intrusion Detection Systems (NIDS) powered by Machine Learning (ML). However, existing ML models often exhibit limited generalizability across diverse network environments. This study bridges this critical gap by comprehensively evaluating the cross-domain performance of six supervised learning algorithms (three shallow, three deep) on four benchmark datasets with distinct attack types, traffic volume, and protocol distribution characteristics. A robust methodology incorporating both in-domain and cross-domain evaluations is employed. In-domain assessments expose the peak performance of each ML algorithm within its training dataset. Conversely, cross-domain evaluations unveil their generalizability by testing performance on unseen datasets. This two-pronged approach provides a nuanced understanding of each algorithm's adaptability and reliability in real-world network scenarios. The findings reveal remarkable in-domain accuracy exceeding 95% for Random Forest and Extra Trees, with Long Short-Term Memory (LSTM) also demonstrating strong performance. However, cross-domain evaluations expose a significant performance decay, with ensemble methods exhibiting an average accuracy drop of 50 % and LSTMs exhibiting a lower average decay of 30.4 %. These results underscore the crucial need for robust cross-domain evaluation in NIDS development.