DISPINV - EFFICIENT DETERMINISTIC LLM SERVING ON HOPPER VIA HOST-DISPATCH PINNING

Deterministic LLM serving is essential for reproducing results, compliance auditing, evaluation andreinforcement-learning rollouts, yet the determinism modes of current engines give up hardware efficiency andprefix-cache reuse. We present DispInv, which achieves determinism, hardware efficiency and prefix-cachesafety on NVIDIA Hopper by pinning host-side dispatch. On a pinned software stack, we find that Hopper’s dy-namic Mixture-of-Experts (MoE) tiles and workspace-constrained warpgroup matrix-multiply-accumulate (WG-MMA) kernels already preserve per-row accumulation order. In the audited shapes, within-process drift comesfrom host-side dispatch decisions that change a row’s reduction tree with batch or cache state: split-K, theswitch to general matrix-vector (GEMV) kernels on narrow MoE routers, and occupancy-driven attention splits.DispInv therefore keeps native kernels, pins narrow routers, injects length-keyed splits into FlashAttention-3metadata and admits a request only when the worst-case KV demand of every running request stays reserved,while DispInv-Control binds these mechanisms to each deployment’s phase, parallelism and capacity contractbefore inference. Across eight dense and MoE models on H200, DispInv shows zero observed drift under batch-ing, caching and preemption perturbations, at 0.92–1.01×the whole-call latency of non-deterministic vLLM atshort context, 0.96–0.98×its decode latency at 32k context and 0.96–1.05×its serving throughput; on Mixtral-8x7B, the narrow-router pin adds 2–8% whole-call latency.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24260690
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

DISPINV - EFFICIENT DETERMINISTIC LLM SERVING ON HOPPER VIA HOST-DISPATCH PINNING

National Yang Ming Chiao Tung University
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
preprint

DISPINV - EFFICIENT DETERMINISTIC LLM SERVING ON HOPPER VIA HOST-DISPATCH PINNING

National Yang Ming Chiao Tung University
preprint en

Abstract

Deterministic LLM serving is essential for reproducing results, compliance auditing, evaluation andreinforcement-learning rollouts, yet the determinism modes of current engines give up hardware efficiency andprefix-cache reuse. We present DispInv, which achieves determinism, hardware efficiency and prefix-cachesafety on NVIDIA Hopper by pinning host-side dispatch. On a pinned software stack, we find that Hopper’s dy-namic Mixture-of-Experts (MoE) tiles and workspace-constrained warpgroup matrix-multiply-accumulate (WG-MMA) kernels already preserve per-row accumulation order. In the audited shapes, within-process drift comesfrom host-side dispatch decisions that change a row’s reduction tree with batch or cache state: split-K, theswitch to general matrix-vector (GEMV) kernels on narrow MoE routers, and occupancy-driven attention splits.DispInv therefore keeps native kernels, pins narrow routers, injects length-keyed splits into FlashAttention-3metadata and admits a request only when the worst-case KV demand of every running request stays reserved,while DispInv-Control binds these mechanisms to each deployment’s phase, parallelism and capacity contractbefore inference. Across eight dense and MoE models on H200, DispInv shows zero observed drift under batch-ing, caching and preemption perturbations, at 0.92–1.01×the whole-call latency of non-deterministic vLLM atshort context, 0.96–0.98×its decode latency at 32k context and 0.96–1.05×its serving throughput; on Mixtral-8x7B, the narrow-router pin adds 2–8% whole-call latency.

Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon