Fault Diagnosis or Experiment Recognition? A Leakage-Aware Evaluation of Machine Learning for Grid-Connected PV Fault Diagnosis

Machine learning classifiers trained on the public GPVS-Faults dataset routinely report near-perfect accuracy in diagnosing faults of grid-connected photovoltaic (PV) systems. We show that this figure measures the ability to recognize which experiment a measurement window comes from rather than which fault is present. Under the commonly used protocol, in which every window of a recording is labeled with that recording’s fault class and windows are split randomly, a random forest reaches 100% accuracy. It reaches the same 100% when it is trained and tested exclusively on data recorded before any fault was injected, i.e., when there is no fault to diagnose. We then evaluate five classifiers under progressively stricter protocols: onset-aware labels, temporally blocked cross-validation with a purge gap, and cross-operating-mode transfer (MPPT/IPPT). Under cross-mode transfer, macro-F1 falls to 0.17-0.32 for all models. Expressing each feature as a deviation from a short healthy reference period of the same installation (baseline referencing) raises the best cross-mode macro-F1 to 0.49. A training-free detector built on the same referenced features, calibrated on MPPT data only, attains a 0.7% false-alarm rate in both modes and detects PV open-circuit, feedback-sensor and inverter faults in 67-99% of post-fault windows, while partial shading and MPPT-controller faults remain undetected. We release the code and evaluation protocol to support reproducible benchmarking.