Adaptive protocols and reconfigurable optical interconnects for datacenter networks

Datacenter networks form the backbone of modern computing and storage, driving the expansion of online services and applications. As these networks evolve, the demand for higher bandwidth and lower latency has increased significantly. Recently, GPU clusters have emerged within datacenters for large-scale distributed training, presenting unique networking challenges. This thesis addresses several critical challenges in datacenter networks --- congestion control, load balancing, buffer sharing, and reconfigurable optical interconnects --- and is structured into three main parts. First, we address the transport protocol problem in modern datacenters. We propose PowerTCP, a novel congestion control algorithm that dynamically adjusts the congestion window based on the bandwidth-window product (or “power”), a new congestion indicator. Through both analytical and empirical validation, we demonstrate PowerTCP’s practicality in datacenter networks, showing that it meets key requirements for high throughput and low latency. We further introduce Ethereal, a transport protocol specifically designed for distributed training workloads in leaf-spine topologies. Ethereal achieves optimal load balancing, akin to packet spraying, by minimally splitting a few application flows while maintaining single-path transport semantics from the network’s perspective. Empirically, we show that Ethereal surpasses state-of-the-art algorithms in collective communication completion times. Second, we tackle the buffer-sharing problem in datacenter switches. We propose ABM, an innovative buffer-sharing algorithm that ensures isolation across different traffic classes while improving burst absorption. As an extension, we introduce Reverie, a solution that enables lossy and lossless traffic to coexist within the same network. Reverie preserves isolation between these traffic types while enhancing burst absorption and flow completion times for both. Additionally, we propose Credence, the first buffer-sharing algorithm to integrate machine-learned predictions. Our analysis shows that Credence achieves near-optimal throughput under perfect predictions and performs effectively even with imperfect predictions, significantly improving flow completion times in empirical tests. Finally, as Moore’s law approaches its limits, we address the challenge of high-performance optical interconnects in datacenter networks. We present the first formal result on the throughput of periodic networks, establishing an equivalence to a corresponding static network. Based on this result, we propose Mars, a demand-oblivious reconfigurable optical interconnect that achieves near-optimal throughput and low latency across various traffic patterns, even with shallow buffers. Additionally, we introduce Vermilion, a demand-aware optical interconnect that dynamically reconfigures according to traffic patterns. Our analysis and empirical results show that Vermilion delivers high throughput across diverse traffic patterns, exceeding the throughput capabilities of demand-oblivious interconnects.