Two Nodes Are Slower Than One: A Pre-registered Study of Data-Parallel Training over 1 GbE, in Which a Two-Parameter Link Model Predicted Every Step Time Within 2.7 Percent
Adding a second GPU node is the standard first step when one machine feels too small for training, and on a slow link it is often the wrong step. This study pre-registered the measurement before running it. Two NVIDIA DGX Spark nodes (one GB10 GPU each), joined by commodity 1 GbE through an office switch, ran data-parallel training (PyTorch DDP, DDP with a bf16 gradient-compression hook, and FSDP2) on GPT-style models of 5.3M, 29.5M, and 91.5M parameters, across gradient-accumulation factors of 1, 4, 16, and 64, against a single-node arm at the same global batch. Seven hypotheses with frozen pass lines were committed before any main-grid run. Five pass. Two nodes are slower than one in 9 of 12 DDP cells, down to a speedup of 0.069 (95 percent bootstrap interval 0.067 to 0.075) for the 91.5M model at accumulation 1, and faster only at accumulation 64 (1.31x to 1.47x). A two-parameter link model, t(S) = alpha + S / beta, fitted on 126 all-reduce trials before any two-node training run (alpha 0.40 ms, beta 117.4 MB/s, R squared 0.9999998), predicted every two-node DDP step time within 2.7 percent, inside its registered 25 percent line, and called the faster-or-slower side correctly in all 12 cells. Correctness is exact, not approximate: after 30 steps, two-node DDP and FSDP2 parameters differ from a single-process replay by 0.0, and crash recovery at steps 13, 25, and 37 of a 40-step run ends bit-identical to the uninterrupted run in all three cases. Two hypotheses missed, narrowly, and are reported as missed: the bf16 hook cut step time by 29.4 percent in one qualifying cell against a 30 percent line (range across qualifying cells 29.4 to 48.7 percent, with the loss condition met everywhere), and FSDP2 peak memory was 65.3 percent of DDP's against a 65 percent line. The claims are deliberately narrow: two nodes, one slow link, short throughput runs. Within those bounds, the crossover is measured, the prediction was locked before the runs it predicts, and no pass line moved.
Authors
- Jesun Ahmad Ushno (ORCID: https://orcid.org/0000-0002-6070-9025)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24328092
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- preprint