Symphony: Collective Coordination in Multi-Tenant GPU Clusters

Multi-tenant GPU clusters are designed to concurrently run multiple distributed ML training workloads. However, frequent data transfers among GPUs via collective operations can slow down training, as collectives from different tenants compete for network bandwidth. Recent work (e.g., CASSINI) has considered collective coordination to prevent network contention, but primarily focused on static job-level optimizations at deployment time, oblivious to runtime network conditions and the specific traffic pattern of each workload. In this paper, we present Symphony, an application-layer solution that dynamically coordinates collective operations across tenants at runtime. Symphony integrates seamlessly with existing clusters with minimal modifications to the collective communication library and includes a lightweight online scheduling mechanism that requires no advance information about the workloads or their collectives. We evaluate Symphony using both a real GPU testbed implementation and trace-driven simulations. Specifically, using realistic ML workloads in our testbed, we observe improvements of up to 13.2% in average communication time and 9.6% in training time compared to state-of-the-art solutions.