SURMA: Efficient Fine-Grained Communication Offload for SuperPOD

GPU SuperPODs underpin large language model (LLM) serving, but growing bandwidth and round-trip times make inter-GPU communication consume increasing compute resources. SM-driven remote memory access occupies shared memory or registers for in-flight data and competes for SM-to-NoC bandwidth. Dedicated engines can free these resources, but must handle fine-grained transfers, down to a 576\,B KV-cache entry. NVIDIA's CopyEngine uses less than 1\% of NVLink bandwidth at 7\,KB granularity. We present SURMA, an on-die Scale-Up Remote Memory Access engine. Adapting RDMA requires overcoming slow doorbell submission, excessive per-application connection state, head-of-line blocking of completion notifications on shared connections, and costly ordering for multipath transfers. SURMA uses a credit-based shared send queue to reduce submission latency from 1.2\,$μ$s to 400\,ns; fused write-with-atomic semantics to share connections while independently notifying each transaction; and Network Per Path Ordering to ensure data arrives before the completion flag without Sender or Receiver Ordering. We implement SURMA in RTL on a two-FPGA prototype and evaluate it with two 8-GPU NVLink servers. At 512\,B granularity, SURMA saturates attainable bandwidth with a single SM, whereas NVLink needs over 14\% of the SMs. SURMA occupies under 0.5\% of GPU area and accelerates inference on four models by 1.05--1.26$\times$ over NVLink.

Authors

Publication Details

Journal
arXiv (Cornell University)
Published
2026-11-16
DOI
https://doi.org/10.48550/arxiv.2611.21274
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

SURMA: Efficient Fine-Grained Communication Offload for SuperPOD

Ran Shu, X. T. Huang, Yachen Wang, Yinben Xia et al.
arXiv (Cornell University)
Parallel Computing and Optimization Techniques
preprint

SURMA: Efficient Fine-Grained Communication Offload for SuperPOD

Ran Shu, X. T. Huang, Yachen Wang, Yinben Xia, Xianneng Zou, Haoran Shang, Sihan Guan, Xinru Tang, Zekun He, Mengzhu Li, Yang Hu, Huizheng Wang, Bowen Liu, Ruoyu Wu, Xiang Li, Weifeng Zhang, Jinyi Deng
preprint en

Abstract

GPU SuperPODs underpin large language model (LLM) serving, but growing bandwidth and round-trip times make inter-GPU communication consume increasing compute resources. SM-driven remote memory access occupies shared memory or registers for in-flight data and competes for SM-to-NoC bandwidth. Dedicated engines can free these resources, but must handle fine-grained transfers, down to a 576\,B KV-cache entry. NVIDIA's CopyEngine uses less than 1\% of NVLink bandwidth at 7\,KB granularity. We present SURMA, an on-die Scale-Up Remote Memory Access engine. Adapting RDMA requires overcoming slow doorbell submission, excessive per-application connection state, head-of-line blocking of completion notifications on shared connections, and costly ordering for multipath transfers. SURMA uses a credit-based shared send queue to reduce submission latency from 1.2\,$μ$s to 400\,ns; fused write-with-atomic semantics to share connections while independently notifying each transaction; and Network Per Path Ordering to ensure data arrives before the completion flag without Sender or Receiver Ordering. We implement SURMA in RTL on a two-FPGA prototype and evaluate it with two 8-GPU NVLink servers. At 512\,B granularity, SURMA saturates attainable bandwidth with a single SM, whereas NVLink needs over 14\% of the SMs. SURMA occupies under 0.5\% of GPU area and accelerates inference on four models by 1.05--1.26$\times$ over NVLink.

arXiv (Cornell University)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon