Do Smart Grid Intrusion Detectors Learn Attacks or Operating Conditions? Evidence from a PMU-Relay Testbed and an IEEE 14-Bus Simulation

Machine learning detectors of cyber attacks on power systems are often evaluated with a random split of samples. We study whether such detectors learn attacks or operating conditions, using a public dataset recorded on a PMU-relay testbed and a 42-day simulation of the IEEE 14-bus system with phasor measurement units and false data injection (FDI). On the testbed data, a random forest separates attacks from natural events with a macro-F1 of 0.90 under a random split but only 0.61 when whole sets of operating conditions are held out, with an accuracy below the majority-class baseline; the sets themselves can be identified with 95.5% accuracy from normal-operation samples. Load-independent physics features, including sequence-component ratios and consistency between the two ends of a line, reduce the gap between the protocols from 0.29 to 0.04 but do not raise the grouped score above 0.63. In the simulation, stealthy FDI constructed to preserve the state-estimation residual bypasses the chi-square bad data detector (1.1% detection). A random forest on PMU measurements and the residual detects 99.9% of stealthy attacks on future days and 86% of attacks on buses not seen during training, at false-alarm rates of 1.4-1.8%, provided that its alarm threshold is calibrated on held-out normal data; with the default threshold it detects 62% despite a ROC-AUC of 0.996. We release the simulation and evaluation code.