Characterizing Modern GPU Resilience and Impact in HPC Systems: A Case Study of A100 GPUs

In this study, we characterize GPU failures in Delta<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>Delta is an HPC system operated by the National Center for Supercomputing Applications (NCSA) at the University of Illinois Urbana-Champaign., a current large-scale AI system with over 1,300 petaflops of peak compute throughput. Using three years (12.5M GPU hours) of GPU error-recovery data, we evaluate the resilience of NVIDIA A100 GPU hardware components to determine the vulnerability of various GPU components to failure and the impact on GPU-node availability and user jobs. Our key findings are as follows. (i) The A100 GPU node resilience in MTBE (mean time between errors) was slightly lower (23%) in the operational period than in the pre-operational period due to more frequent GPU hardware errors, primarily caused by higher GPU utilization. (ii) Contrary to common belief, GPU memory is 160x more reliable than GPU hardware in terms of MTBE. (iii) The newly introduced GSP (GPU system Processor) is the most vulnerable GPU hardware component. (iv) NVLink errors did not always lead to user job failures; we attribute this in part to the underlying error detection and retry mechanisms. (v) Hardware errors and insufficient recovery mechanisms cause frequent job failures and limit overall GPU node availability to 99.5% (7 minutes of downtime per day), indicating that the infrastructure is not yet ready for system-scale, long-running user jobs.