Evaluating Machine Learning Techniques for ICS Security: Insights from Dataset Limitations and Classifier Performance

The security of Industrial Control Systems (ICSs) is critical for the operation of essential infrastructure like energy, water, and transportation. However, ICS communication protocols often lack essential security features, making them vulnerable to advanced persistent threat (APT) attacks, as evidenced by cyberattacks like Stuxnet and Triton. One potential solution is deploying Intrusion Detection Systems (IDS) to monitor network traffic and detect intrusions. While Machine Learning (ML) and Deep Learning (DL) have shown promise for IDS, testing these methods in real ICS environments is impractical due to the risk of disruptions and the challenge of labeling data without altering system functionality. This study compares the performance of several ML classifiers, including Logistic Regression, Decision Tree, XGBoost, Random Forest, ANN, LightGBM, and SVM, in detecting ICS cyberattacks. The classifiers generally performed well but showed variations depending on the type of attack. For instance, Decision Tree, Random Forest, and SVM excelled in detecting DDoS attacks, while performance dropped for PortScan attacks. LightGBM outperformed others across multiple attack types, with F-scores between 0.993 and 1.000. The study highlights the importance of comprehensive, labeled datasets for improving IDS effectiveness in ICS environments.