Compute-based Fault Tolerance for DNN

Deep neural network (DNN) systems use many GPUs, which can fail---making fault tolerance (FT) essential to avoid cluster restarts. Traditional FT relies on frequent checkpointing, incurring high bandwidth and memory costs. We propose an alternative strategy using GPU redundancy, introducing uniform and heterogeneous encoding approaches. We analyze their costs and recommend usage scenarios, especially for emerging rack-scale AI computers