GC-059 Large-Scale Cybersecurity Threat Detection

Cybersecurity threats are becoming more sophisticated, posing serious risks to critical systems. Traditional intrusion detection systems often fail to manage the scale and complexity of network traffic. This study investigates large-scale threat detection using machine learning in PySpark, utilizing the UNSW-NB15 dataset. It focuses on building scalable models through preprocessing, feature selection, and implementing algorithms like Decision Trees, Naïve Bayes, Random Forest, and Gradient Boosting. Evaluation metrics include accuracy, precision, recall, F1-score, and ROC-AUC, with emphasis on hyperparameter tuning and minimizing false positives. Leveraging PySpark’s distributed computing, the system ensures efficient real-time analysis of vast network data. The research supports modern cybersecurity strategies by enhancing detection reliability and reducing risks from emerging cyber threats.