Two Nodes Are Slower Than One: A Pre-registered Study of Data-Parallel Training over 1 GbE, in Which a Two-Parameter Link Model Predicted Every Step Time Within 2.7 Percent

Adding a second GPU node is the standard first step when one machine feels too small for training, and on a slow link it is often the wrong step. This study pre-registered the measurement before running it. Two NVIDIA DGX Spark nodes (one GB10 GPU each), joined by commodity 1 GbE through an office switch, ran data-parallel training (PyTorch DDP, DDP with a bf16 gradient-compression hook, and FSDP2) on GPT-style models of 5.3M, 29.5M, and 91.5M parameters, across gradient-accumulation factors of 1, 4, 16, and 64, against a single-node arm at the same global batch. Seven hypotheses with frozen pass lines were committed before any main-grid run. Five pass. Two nodes are slower than one in 9 of 12 DDP cells, down to a speedup of 0.069 (95 percent bootstrap interval 0.067 to 0.075) for the 91.5M model at accumulation 1, and faster only at accumulation 64 (1.31x to 1.47x). A two-parameter link model, t(S) = alpha + S / beta, fitted on 126 all-reduce trials before any two-node training run (alpha 0.40 ms, beta 117.4 MB/s, R squared 0.9999998), predicted every two-node DDP step time within 2.7 percent, inside its registered 25 percent line, and called the faster-or-slower side correctly in all 12 cells. Correctness is exact, not approximate: after 30 steps, two-node DDP and FSDP2 parameters differ from a single-process replay by 0.0, and crash recovery at steps 13, 25, and 37 of a 40-step run ends bit-identical to the uninterrupted run in all three cases. Two hypotheses missed, narrowly, and are reported as missed: the bf16 hook cut step time by 29.4 percent in one qualifying cell against a 30 percent line (range across qualifying cells 29.4 to 48.7 percent, with the loss condition met everywhere), and FSDP2 peak memory was 65.3 percent of DDP's against a 65 percent line. The claims are deliberately narrow: two nodes, one slow link, short throughput runs. Within those bounds, the crossover is measured, the prediction was locked before the runs it predicts, and no pass line moved.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328092
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Two Nodes Are Slower Than One: A Pre-registered Study of Data-Parallel Training over 1 GbE, in Which a Two-Parameter Link Model Predicted Every Step Time Within 2.7 Percent

Jesun Ahmad Ushno
3 links
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
preprint

Two Nodes Are Slower Than One: A Pre-registered Study of Data-Parallel Training over 1 GbE, in Which a Two-Parameter Link Model Predicted Every Step Time Within 2.7 Percent

Jesun Ahmad Ushno
preprint en

Abstract

Adding a second GPU node is the standard first step when one machine feels too small for training, and on a slow link it is often the wrong step. This study pre-registered the measurement before running it. Two NVIDIA DGX Spark nodes (one GB10 GPU each), joined by commodity 1 GbE through an office switch, ran data-parallel training (PyTorch DDP, DDP with a bf16 gradient-compression hook, and FSDP2) on GPT-style models of 5.3M, 29.5M, and 91.5M parameters, across gradient-accumulation factors of 1, 4, 16, and 64, against a single-node arm at the same global batch. Seven hypotheses with frozen pass lines were committed before any main-grid run. Five pass. Two nodes are slower than one in 9 of 12 DDP cells, down to a speedup of 0.069 (95 percent bootstrap interval 0.067 to 0.075) for the 91.5M model at accumulation 1, and faster only at accumulation 64 (1.31x to 1.47x). A two-parameter link model, t(S) = alpha + S / beta, fitted on 126 all-reduce trials before any two-node training run (alpha 0.40 ms, beta 117.4 MB/s, R squared 0.9999998), predicted every two-node DDP step time within 2.7 percent, inside its registered 25 percent line, and called the faster-or-slower side correctly in all 12 cells. Correctness is exact, not approximate: after 30 steps, two-node DDP and FSDP2 parameters differ from a single-process replay by 0.0, and crash recovery at steps 13, 25, and 37 of a 40-step run ends bit-identical to the uninterrupted run in all three cases. Two hypotheses missed, narrowly, and are reported as missed: the bf16 hook cut step time by 29.4 percent in one qualifying cell against a 30 percent line (range across qualifying cells 29.4 to 48.7 percent, with the loss condition met everywhere), and FSDP2 peak memory was 65.3 percent of DDP's against a 65 percent line. The claims are deliberately narrow: two nodes, one slow link, short throughput runs. Within those bounds, the crossover is measured, the prediction was locked before the runs it predicts, and no pass line moved.

Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon