Dataset Separability Versus Model Architecture in Electricity Theft Detection: A Rigorous Comparison of Four Machine Learning Models Across Two Datasets

Electricity theft leads to large amounts of non-technical losses for utility companies. Although the reported detection performance differs considerably, the various studies have used different algorithms, datasets and pipelines, so the variation cannot be assigned to any one factor. This study investigates what proportion of the difference is due to the architecture and what proportion is due to the dataset and the way in which it is labeled. The four methods, Random Forest, XGBoost, LightGBM and Isolation Forest, are compared using the same leakage-controlled pipeline with two datasets: the SGCC benchmark, which has labels derived from field inspections, and half-hourly consumption data from 53 schools in the Western Cape that has been subjected to injected suppression attacks. On the SGCC dataset, the supervised models achieve a theft class F1 score of 0.366 to 0.403 at accuracy levels of 0.88 to 0.90, with no significant pairwise differences (p = 0.126–0.867). With respect to the Western Cape dataset, XGBoost performs best (F1 0.727, FNR 0.294), once again without significance. Since there are only 17 positive cases, the smallest detectable difference in FNR is 0.450, and therefore these tests are uninformative about architecture rather than evidence of equivalence. Isolation Forest does worse on both datasets (ROC-AUC 0.640 and 0.565). None of the SGCC models outperforms the 91.47% majority baseline. When evaluated in the same score space, the binormal separability index increases from 1.203 to 2.060. The only F1 effect that is supported by the confidence intervals is the cross-dataset gap of +0.157 to +0.325. The reported differences are therefore due to the dataset and its labeling procedure, not to the architecture.