Mirror, Mirror on the Wall, What is the Best Topology of Them All?

Mirror, Mirror on the Wall, What is the Best Topology of Them All?AI is one of the most important emerging technologies of the 21 st century, and designing suitable infrastructure for large-scale AI systems is critical.Major companies such as Microsoft, Google, Meta, and even Tesla are touting large-scale "AI supercomputers" as an essential tool for increasingly powerful AI systems, but as AI systems have a more specialized workload than traditional supercomputers, designing and implementing their architecture is a complex process.The complexity is because AI systems are specialized for AI and machine learning (ML) workloads, leveraging parallelism and specialized hardware accelerators to excel in processing and analyzing large datasets to make predictions, classify objects, and understand natural language.Traditional supercomputers, on the other hand, are general-purpose machines used for a broader range of scientific and computational tasks.The authors of the accompanying paper leverage the unique demands of specialized AI workloads to craft a network structure tailored for large-scale deep learning, which is a pivotal facet of AI.The authors describe AI workloads by considering three dimensions of parallelism: data parallelism, pipeline parallelism, and operator parallelism.Data parallelism is used in AI workloads when training ML models.Pipeline parallelism can be observed in AI workloads in the execution of complex neural network models.In AI workloads, operator parallelism can be seen in the parallel execution of mathematical operations that make up neural network layers.The paper argues that, although each dimension is different, it can be implemented with nearest-neighbor communication, but today's high-performance computing (HPC) networks often overprovision global bandwidth and underprovision local bandwidth for AI workloads.This insight motivates the authors to use toroidal networks that HPC has been using traditionally but abandoned in favor of more flexible low-diameter topologies based on switching technologies.Torus network topologies extend the torus concept into multiple dimensions.These networks offer efficient connectivity between processing nodes.Google's early Tensor Processing Units (TPUs) also employed two and three-dimensional torus networks for interconnection.This architecture facilitates efficient communication between TPUs in datacenters, which is crucial for ML workloads.While torus networks offer advantages like low latency and determinism, low-diameter torus topologies can suffer from limited global bandwidth.Scheduling and managing traffic on torus networks can be inflexible, leading to performance bottlenecks.More flexible switch technologies, such as the Dragonfly network, have gained popularity in HPC to address the limitations of low-diameter torus networks.Switched topologies generally use switches to route data efficiently between networked devices.Switched networks use switches as