Predictive GPU Failure Detection via Continuous Health Scoring: A Reproducible Study on Synthetic Fleet Telemetry

In large accelerator fleets, GPU hardware failure is a routine operational event rather than an exception, and a failure that strikes mid-run on a long, stateful training job is costly: the accumulated computation is lost. Conventional reliability tooling treats a GPU as binary — healthy or failed — and alerts only when a hard threshold is crossed, by which point the device is often already failing. This paper studies an alternative: treating each GPU as a stateful entity with a continuous health score that estimates near-term failure risk from telemetry trends, so that at-risk work can be drained proactively. Because production failure data is proprietary and scarce, we develop an open, reproducible synthetic telemetry model grounded in publicly documented precursor behavior (rising ECC correctable-error rates, power-draw deviation, thermal-margin erosion, and interconnect error growth), including realistic complications — benign transient spikes and a fraction of abrupt, precursor-free failures. On a held-out set of 500 simulated GPUs (37 failures), a health score built from exponentially-weighted trend and self-baseline-deviation features achieves a precision-recall AUC of 0.78 and, at a fixed 90% precision operating point, warns a median of 24 hours before failure while catching 81% of failures. A reactive baseline — a static threshold on the raw ECC rate tuned to the same precision — warns only a median of 2 hours ahead and catches 54%. The health score does not detect more of the abrupt, precursor-free failures; its advantage is earlier warning on the detectable ones, which is precisely what enables proactive migration. The full simulator, model, and evaluation are released as an open-source reference implementation. All results are from synthetic data and are intended to demonstrate the method's mechanics, not to quantify real-world accuracy.