Enhancing IDS performance through a comparative analysis of Random Forest, XGBoost, and Deep Neural Networks

Intrusion Detection Systems (IDS) face major challenges in network security, notably the need to combine a high detection rate with reliable performance. This reliability is often affected by class imbalances and inadequate hyperparameter optimization. This article addresses the issue of improving the detection rate of IDS by evaluating and comparing three machine learning algorithms: Random Forest (RF), XGBoost, and Deep Neural Networks (DNN), using the NSL-KDD dataset. In our methodology, we integrate SMOTE (Synthetic Minority Oversampling Technique) to tackle the unbalanced nature of the data, ensuring a more balanced representation of the different classes. This approach helps optimize model performance, reduce bias, and enhance robustness. Additionally, hyperparameter optimization is performed using Optuna, ensuring that each algorithm operates at its optimal level. The results show that our model, using the Random Forest algorithm, achieves an accuracy of 99.80%, surpassing the performance of XGBoost and Deep Neural Networks (DNN). This makes our approach a true asset for intrusion detection methods in computer networks. • Random Forest achieved 99.80% accuracy and 0.9988 AUC on the NSL-KDD dataset. • Comparative performance evaluation of RF, XGBoost, and DNN for IDS. • Data imbalance in IDS addressed using the SMOTE technique. • The proposed approach outperforms the most comparable IDS models in detection accuracy. • IDS framework adaptable to various network datasets.