Multi-Stage Approach for Identifying Defective Hardware in Frontier

In June 2022, the long-awaited exaflop compute barrier (1 quintillion floating-point operations per second) was surpassed on the TOP500 list by Frontier, an HPE Cray EX supercomputer at Oak Ridge National Laboratory (ORNL).Drawing peak power of 21.1 MW, Frontier demonstrated 1.1 exaflops/second of computational capability, much of which is supplied by more than 37,000 AMD Instinct MI250X graphics processing units (GPUs).With a single GPU drawing up to 560W thermal design power (TDP), each AMD MI250X draws 2x more power under load than the NVIDIA V100 GPUs used in Frontier's predecessor at ORNL, the 200 petaflop IBM POWER9 supercomputer, Summit.There are many other major technological advances in the memory, compute, power, and infrastructure of Frontier that are new to production environments.Frontier's mission to enable ground-breaking research in U.S. energy, economic and national security sectors is fulfilled through leadership-class workloads, which are workloads that demand greater than 20% of the supercomputer.These large workloads are vulnerable to defective and failing computing hardware.The rate of failing hardware is quantified through the mean time between failures (MTBF), which is the length of time between a hardware-level failure anywhere in the system.In this work, we describe the multi-staged approaches to stabilizing and maintaining the functionality of the hardware on Frontier.There are three strategies discussed; the first two utilize leadership-class tests to target improving the MTBF of Frontier, the third utilizes single-node validation to efficiently identify individual instances of defective hardware in Frontier.We provide summarized data from each of the three strategies, then classify the diverse set of failures and discuss trends in defective hardware, before discussing several key challenges to identifying defective hardware and improving the MTBF of Frontier.