1242-GRP Distributed Cycle Splitter

Deep learning systems often permute tensors distributed across GPUs. PyTorch's all_to_all_nd writes each GPU's result into a separate output buffer, doubling the memory each GPU's share occupies. We design Distributed Cycle Splitter (DCS), an in-place alternative to all_to_all_nd for the same permutations. DCS keeps its total auxiliary memory within 5% of the tensor instead of a full output buffer. It follows the permutation's cycles, writes blocks that must move across GPUs into the other GPU's inbox buffer through symmetric memory, and the receiver unpacks them one group at a time. On two PCIe-connected GPUs with 40 MB to 4 GB tensors, DCS stays within this bound on all 271 shapes and uses 94-98% less auxiliary memory per GPU than all_to_all_nd.

Authors

Publication Details

Journal
DigitalCommons - Kennesaw State University (Kennesaw State University)
Published
2026-11-20
Primary Topic
Parallel Computing and Optimization Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

1242-GRP Distributed Cycle Splitter

Xuechen ; Zhang, Kaveen Jayamanna
DigitalCommons - Kennesaw State University (Kennesaw State University)
Parallel Computing and Optimization Techniques
article

1242-GRP Distributed Cycle Splitter

Xuechen ; Zhang, Kaveen Jayamanna
article en

Abstract

Deep learning systems often permute tensors distributed across GPUs. PyTorch's all_to_all_nd writes each GPU's result into a separate output buffer, doubling the memory each GPU's share occupies. We design Distributed Cycle Splitter (DCS), an in-place alternative to all_to_all_nd for the same permutations. DCS keeps its total auxiliary memory within 5% of the tensor instead of a full output buffer. It follows the permutation's cycles, writes blocks that must move across GPUs into the other GPU's inbox buffer through symmetric memory, and the receiver unpacks them one group at a time. On two PCIe-connected GPUs with 40 MB to 4 GB tensors, DCS stays within this bound on all 271 shapes and uses 94-98% less auxiliary memory per GPU than all_to_all_nd.

DigitalCommons - Kennesaw State University (Kennesaw State University)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
1242-GRP Distributed Cycle Splitter — Xuechen ; Zhang, Kaveen Jayamanna · DigitalCommons - Kennesaw State University (Kennesaw State University) (2026) | TGRS Research Map | TGRS