What Does a Streaming Client See When an LLM Server Crashes? A Fault-Injection Study of Client-Visible Token Anomalies in vLLM and SGLang

LLM inference servers stream generated tokens to clients over requests that last seconds to minutes. When a serving process fails mid-generation, the client is left holding a partial response, and the recovery it performs determines what it ultimately shows a user or hands to a downstream agent. We built an open-source fault-injection harness that SIGKILLs an unmodified OpenAI-compatible server at a chosen stream chunk, restarts it, applies a client recovery strategy, and records every streamed chunk, and applied it to vLLM and SGLang serving Llama 3.1 8B Instruct on A10G GPUs (86 runs, 430 trials including ablations and follow-ups). In both engines' default configurations the failure-free output for a fixed prompt was not unique at temperature 0, for different reasons: in vLLM it depended deterministically on what the server had served before, through automatic prefix caching; in SGLang the first request after each start varied from start to start. Because a crashed server restarts cold, naive retry contradicted text the client had already displayed in 25 of 30 vLLM and 24 of 30 SGLang trials on the main prompt, and in at least one trial on 3 of 5 (vLLM) and 4 of 5 (SGLang) further prompts. Recompute, which resubmits the delivered text, never contradicted it but usually changed the final answer. Re-tokenization was ruled out as the cause. With each engine's deterministic mode enabled (vLLM batch invariance, SGLang deterministic inference), recompute reproduced the failure-free output exactly in every temperature-0 trial we ran (35 of 35; on vLLM also at temperature 0.8), and naive retry never contradicted the client at temperature 0. This is failure-free equivalence for sequential requests and fail-stop crashes, not an exactly-once protocol. Recovery latency was dominated by server restart (107-195 s). All raw logs and the table-generating script are public.