Draft-Model Speculative Decoding Can Emit Zero-Probability Tokens

Abstract Speculative sampling is supposed to leave the target model's output distribution unchanged. We test that claim in five inference engines by comparing each speculative path with the same engine run without speculation. On a tiny fixture we enumerate every output sequence exactly, so the test sees probabilities, not just the greedy path. TensorRT-LLM 1.2.1's two-model draft-target path, run with its VANILLA attention backend, fails at every sampling setting we ran. Under top-k sampling, 45% of its sampled sequences have probability zero under the engine's own top-k distribution, while greedy decoding still matches. The verifier receives fewer draft-probability rows than draft tokens, and in context-start rounds the one row it gets belongs to the next token. Forty-seven cells across vLLM, SGLang, llama.cpp and Hugging Face Transformers show no deviation at the stated detection limits, and neither do EAGLE-3 chain drafters in vLLM and SGLang in a supplementary test. Data and code The preregistration, harness, raw draws and analysis scripts are in https://github.com/SeverinVisionary/spec-decode-exactness at commit ece0fd3. Disclosure This paper is AI-assisted. I designed the study and decided every claim. Claude Opus 5.5 (Anthropic) wrote the harness and the analysis scripts under my direction, and drafted the amendments to the preregistration, the upstream issue and the description of the patch. Gemini 4 (Google) wrote the first version of the upstream fix patch and revised it. Codex and ChatGPT (OpenAI) proposed the supplementary extensions, from which I chose. The preregistration, the harness and this paper went through reviews with Codex, ChatGPT, Gemini and Claude. The prose was drafted with Claude from my notes, and I reviewed and approved it. I am responsible for the content.