LIBRA: Multi-Path Transport in Distributed Training Data Center Network

Effective networking is crucial for the efficient and cost-effective training of artificial intelligence (AI) models. Remote Direct Memory Access (RDMA) has gained widespread adoption due to its low latency, high throughput, and minimal CPU overhead. However, current RDMA implementations remain constrained by underlying Equal-Cost Multi-Path (ECMP) forwarding mechanisms, hindering the full exploitation of parallel paths within distributed training data centers. Furthermore, the rapid evolution of network interface bandwidth necessitates the re-evaluation of existing multi-path approaches due to ineffective scaling to high-speed network environments.This study introduces LIBRA, a novel multi-path transmission method for RDMA within high-speed network environments, enabling efficient utilization of abundant network paths in modern data centers. LIBRA integrates segment routing with packet-level multi-path spraying. Deterministic path assignment eliminates the need for sender-side path detection, simplifying load-balancing design and enhancing overall system efficiency. Furthermore, novel algorithms are proposed to decouple multi-path load balancing from congestion control (CC). These algorithms identify network congestion positions and implement tailored traffic control strategies through fine-grained control mechanisms. The feasibility of the hardware implementation for LIBRA was experimentally validated using FPGA and Tofino-based prototypes. Large-scale simulations demonstrate that LIBRA effectively and robustly leverages the rich network multi-paths within data centers, yielding a 2~4× reduction in tail latency for large message transmission scenarios. Our simulation code will be released as open source at the time of publication.