Traffic Management in Direct Interconnect Data Centers: A Machine Learning Based Approach

Traditional data center architectures, such as Clos-based topologies, can sometimes encounter challenges with scalability and latency. The Direct Interconnect Data Center (DIDC) architecture offers a solution to these issues by enabling direct, low-latency MEMS-based optical connections between servers. This architecture is gaining popularity among web scalers, particularly in data centers that handle low-latency workloads. We introduce a novel machine learning-based algorithm, JIRO, for optimal traffic management in DIDC, considering the unique constraints of this architecture. During the development of this algorithm, we created new techniques for handling integer constraints within the Pytorch gradient descent framework, which can also be applied to other combinatorial optimization problems. Through extensive experiments and comparisons to lower bounds, we demonstrate that JIRO offers performance guarantees for multiple traffic patterns without significant over-provisioning.