Advancing AI-Driven Network Anomaly Detection: A Comparative Study Employing Big Data Analytics

In the expanding realms of the Internet and IoT, the surge in network data both drives the digital economy and intensifies cybersecurity vulnerabilities. Network anomaly detection is essential for protecting against security threats. This paper conducted a comprehensive comparative study by applying big data analytics and sophisticated machine learning to enhance intelligent network anomaly detection. It confronts challenges such as data heterogeneity and model standardization, conducting extensive experiments across six datasets with a range of algorithms, from classical decision trees to cutting-edge CNN, LSTM, and Transformer models, the GWO algorithm was also employed for feature selection, and it was combined with the KNN algorithm to optimize classification performance. The evaluation focuses on metrics such as accuracy, recall rate, F1 score, and training time, revealing the performance of these algorithms with high-dimensional and imbalanced data. Notably, random forest shows exceptional detection performance on the CIC-MalMem-2022 dataset, while random forest and GBDT excel in accuracy and training speed on the RT-IoT2022 dataset. These insights are critical for creating more effective detection systems.