SURMA: Efficient Fine-Grained Communication Offload for SuperPOD
GPU SuperPODs underpin large language model (LLM) serving, but growing bandwidth and round-trip times make inter-GPU communication consume increasing compute resources. SM-driven remote memory access occupies shared memory or registers for in-flight data and competes for SM-to-NoC bandwidth. Dedicated engines can free these resources, but must handle fine-grained transfers, down to a 576\,B KV-cache entry. NVIDIA's CopyEngine uses less than 1\% of NVLink bandwidth at 7\,KB granularity. We present SURMA, an on-die Scale-Up Remote Memory Access engine. Adapting RDMA requires overcoming slow doorbell submission, excessive per-application connection state, head-of-line blocking of completion notifications on shared connections, and costly ordering for multipath transfers. SURMA uses a credit-based shared send queue to reduce submission latency from 1.2\,$μ$s to 400\,ns; fused write-with-atomic semantics to share connections while independently notifying each transaction; and Network Per Path Ordering to ensure data arrives before the completion flag without Sender or Receiver Ordering. We implement SURMA in RTL on a two-FPGA prototype and evaluate it with two 8-GPU NVLink servers. At 512\,B granularity, SURMA saturates attainable bandwidth with a single SM, whereas NVLink needs over 14\% of the SMs. SURMA occupies under 0.5\% of GPU area and accelerates inference on four models by 1.05--1.26$\times$ over NVLink.
Authors
- Ran Shu (ORCID: https://orcid.org/0000-0002-2021-4917)
- X. T. Huang (ORCID: https://orcid.org/0000-0002-4830-7808)
- Yachen Wang (ORCID: https://orcid.org/0009-0003-3669-1202)
- Yinben Xia (ORCID: https://orcid.org/0009-0005-2816-9777)
- Xianneng Zou (ORCID: https://orcid.org/0009-0005-5370-8162)
- Haoran Shang (ORCID: https://orcid.org/0009-0004-6972-9235)
- Sihan Guan (ORCID: https://orcid.org/0009-0001-0282-6039)
- Xinru Tang (ORCID: https://orcid.org/0009-0004-6038-3709)
- Zekun He
- Mengzhu Li
- Yang Hu
- Huizheng Wang
- Bowen Liu
- Ruoyu Wu
- Xiang Li
- Weifeng Zhang
- Jinyi Deng
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.48550/arxiv.2611.21274
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- preprint