A Leakage-Controlled Evaluation Protocol for Machine-Learning Intrusion Detection on the TON_IoT Network Dataset

Intrusion detectors trained on the TON_IoT network data routinely report near-perfect scores, but the extent to which these scores reflect attack behaviour rather than dataset-specific collection artefacts remains unclear. This study evaluates four classifiers using four progressively stricter evaluation protocols on the 211,043-flow train–test file: naive random splitting with all columns, removal of addressing and free-text identifiers, deduplication before splitting, and holding out entire attacker hosts. A probe trained only on IP addresses and ports reached a binary macro-F1 of 0.9999 and a ten-class macro-F1 of up to 0.932, indicating that the identifiers alone are highly predictive of the labels in this dataset. Exact duplicates made up 9.7% of rows, and 12.1% of test rows under naive cross-validation had a copy in the training fold. Removing identifiers cut multiclass macro-F1 from 0.986 to 0.953 for random forest, with man-in-the-middle F1 falling from 0.903 to 0.757. Under host hold-out, attack recall stayed above 0.95, while false-alarm rates on unseen benign hosts varied substantially across hold-outs. The evaluation protocols, identifier-only probe, and reporting checklist provide practical safeguards for future studies using this dataset.