Anomaly Detection in Network Security: A Comparative Study of Cybersecurity Intrusion Detection Machine Learning Algorithms

Introduction: The growing complexity of cyberattacks has made machine learning (ML) algorithms for effective intrusion detection in network security. This study gives a comparative assessment of different supervised and unsupervised ML models, such as DT, RF, SVM, and NN, in terms of testing their efficiency for anomaly detection. The existing literature highlights the efficacy of Random Forest and XGBoost in achieving high classification accuracy, while deep neural networks have demonstrated superior performance in handling complex datasets. With these advances, there are various challenges such as high false positives, computational inefficiency, and class imbalance remain prevalent. The proposed methodology includes rigorous preprocessing of the dataset, feature selection, and model optimisation through hyperparameter tuning to improve the performance of intrusion detection models. Precision value, calculated recall value, F1 score, and AUC-ROC curve are used to determine the performance of the algorithms, however the RF model achieve the highest AUC-ROC value of 88.99%. The results shows that while ensemble models perform better in overall other models, further improvements in feature selection and real-time adaptability are required for enhanced cybersecurity. Objectives: The object of research article is to conduct a comparative study of various types of supervised and unsupervised machine learning algorithms to identify the anomalies within the network traffic. Results: In this work, an extensive comparative analysis was carried out on supervised and unsupervised machine learning models to analyze their performance in anomaly-based intrusion detection in network security. Various models such as Decision Tree, Optimized Decision Tree, SVM, SVM with RBF kernel, Random Forest, and Neural Network were evaluated on a standardized dataset. The models were evaluated in terms of important measures such as accuracy, precision, recall, F1-score, and AUC-ROC. The observations showed that ensemble learning methods, especially Random Forest, had superior classification performance with the best AUC-ROC value of 88.99%. This reflects that ensemble approaches are better suited to identify complex, non-linear patterns of attacks in network traffic. SVM with RBF kernel also showed significant improvements in performance with respect to its linear version, reflecting the advantages of kernelized transformations to address non-linear separability of data. Conclusions: In this work, an extensive comparative analysis was carried out on supervised and unsupervised machine learning models to analyze their performance in anomaly-based intrusion detection in network security. The Neural Network model presented competitive accuracy and AUC-ROC scores, it was limited by high computational overhead and the requirement to balance hyperparameters.