DISPINV - EFFICIENT DETERMINISTIC LLM SERVING ON HOPPER VIA HOST-DISPATCH PINNING
Deterministic LLM serving is essential for reproducing results, compliance auditing, evaluation andreinforcement-learning rollouts, yet the determinism modes of current engines give up hardware efficiency andprefix-cache reuse. We present DispInv, which achieves determinism, hardware efficiency and prefix-cachesafety on NVIDIA Hopper by pinning host-side dispatch. On a pinned software stack, we find that Hopper’s dy-namic Mixture-of-Experts (MoE) tiles and workspace-constrained warpgroup matrix-multiply-accumulate (WG-MMA) kernels already preserve per-row accumulation order. In the audited shapes, within-process drift comesfrom host-side dispatch decisions that change a row’s reduction tree with batch or cache state: split-K, theswitch to general matrix-vector (GEMV) kernels on narrow MoE routers, and occupancy-driven attention splits.DispInv therefore keeps native kernels, pins narrow routers, injects length-keyed splits into FlashAttention-3metadata and admits a request only when the worst-case KV demand of every running request stays reserved,while DispInv-Control binds these mechanisms to each deployment’s phase, parallelism and capacity contractbefore inference. Across eight dense and MoE models on H200, DispInv shows zero observed drift under batch-ing, caching and preemption perturbations, at 0.92–1.01×the whole-call latency of non-deterministic vLLM atshort context, 0.96–0.98×its decode latency at 32k context and 0.96–1.05×its serving throughput; on Mixtral-8x7B, the narrow-router pin adds 2–8% whole-call latency.
Authors
- National Yang Ming Chiao Tung University
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24260690
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- preprint