Reconfigurability within Collective Communication Algorithms

With the explosive growth of AI applications, models and datasets continue to evolve rapidly. Today's Deep neural networks (DNN) consist of billions of parameters, requiring them to utilize many high-performance neural processing units (NPUs, such as GPUs or TPUs). Many large cloud providers use Torus topology as the interconnect for these large-scale training workloads (Google TPU, AWS Trainium, etc.). Resource allocations to multiple tenants in a Torus network can result in topology slices of different sizes, and model training on these torus slices will often result in 1D, 2D, or 3D logical communication. In this paper, we first analyze the performance of state-of-the-art collective communication algorithms on various sizes of static torus topology slices. Then, we quantify the benefits that dynamically reconfigurable torus slices can bring to the state-of-the-art collective communication algorithms.