LSTM-Based Anomaly Detection for Industrial Control Systems on the HAI Testbed: A Comparison of Evaluation Protocols and Metrics

Published results for process-level anomaly detection on the HIL-based Augmented ICS (HAI) dataset range from below 0.3 to above 0.95. This extended version of a conference paper, which reported an accuracy of 0.9996 for a Long Short-Term Memory (LSTM) detector on HAI 20.07, measures how much of this range is due to the evaluation protocol and metric rather than to the detectors. The same trained models are evaluated changing one choice at a time: input definition, data split, class imbalance treatment, threshold selection, point adjustment and evaluated segment. In the conference setting, a rule that copies the previous label scores as high as the LSTM, and a random split of the samples alone yields near-perfect scores. On HAI 23.05, a test-tuned threshold and point adjustment double the reported score of the same detector and compress the differences among six detectors, and random anomaly scores obtain a higher point-adjusted F1 than all of them. Under deployment-style protocols, the best semi-supervised detectors outperform supervised detectors trained on attacks against a subset of the control loops, but none meet a budget of one false alarm per day on the test data. A checklist of evaluation choices, code and data splits are released.

Authors

Institutions

Publication Details

Journal
Computers
Published
2026-11-18
DOI
https://doi.org/10.3390/computers15110850
Primary Topic
Network Security and Intrusion Detection
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

LSTM-Based Anomaly Detection for Industrial Control Systems on the HAI Testbed: A Comparison of Evaluation Protocols and Metrics

Damián Martínez Giracca, Fabio López Pires
3 links
Computers
Network Security and Intrusion Detection
article

LSTM-Based Anomaly Detection for Industrial Control Systems on the HAI Testbed: A Comparison of Evaluation Protocols and Metrics

Damián Martínez Giracca, Fabio López Pires
article en

Abstract

Published results for process-level anomaly detection on the HIL-based Augmented ICS (HAI) dataset range from below 0.3 to above 0.95. This extended version of a conference paper, which reported an accuracy of 0.9996 for a Long Short-Term Memory (LSTM) detector on HAI 20.07, measures how much of this range is due to the evaluation protocol and metric rather than to the detectors. The same trained models are evaluated changing one choice at a time: input definition, data split, class imbalance treatment, threshold selection, point adjustment and evaluated segment. In the conference setting, a rule that copies the previous label scores as high as the LSTM, and a random split of the samples alone yields near-perfect scores. On HAI 23.05, a test-tuned threshold and point adjustment double the reported score of the same detector and compress the differences among six detectors, and random anomaly scores obtain a higher point-adjusted F1 than all of them. Under deployment-style protocols, the best semi-supervised detectors outperform supervised detectors trained on attacks against a subset of the control loops, but none meet a budget of one false alarm per day on the test data. A checklist of evaluation choices, code and data splits are released.

ComputersVol. 15(11)
Universidad Nacional del Este (PY)
Network Security and Intrusion Detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon