Does Low-Communication Training Survive Bad Networks? A Reproducible Study of DDP, DiLoCo and Hierarchical Aggregation over Emulated WAN Links

Training neural networks across geographically distributed sites is limited by slow, lossy wide-area links. Low-communication methods such as DiLoCo synchronize rarely, and hierarchical schemes aggregate within regions before crossing the WAN. We present a reproducible, CPU-scale study that trains a small transformer language model with four workers in two emulated regions and compares synchronous data parallelism (DDP), DiLoCo, and a two-level hierarchical DiLoCo, each with FP32, INT8 or INT4 compression of everything that crosses the WAN. Synchronization rounds are measured over real TCP connections through a packet-level emulator with per-region full-duplex links, finite buffers and a 1500-byte MTU, under four WAN profiles from 100 Mbit/s to 2 Mbit/s with 1% loss. With the same 40 WAN synchronizations, hierarchical DiLoCo reached a validation loss of 1.665 versus 1.696 for flat DiLoCo, and it also beat flat DiLoCo synchronizing five times as often (1.689). INT8 and INT4 compression changed final loss by at most 0.038. DDP reached the best loss (1.587) but, on a 10 Mbit/s link, would spend an estimated 186 minutes on 2,000 steps versus 5.4 minutes for hierarchical DiLoCo with INT8. A real four-process distributed run matched the simulation's validation loss to within 0.031, and the wall-clock model underestimated its total run time by 8-15%. Code and data are released.