PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding

Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification.However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization.We present PipeSpec, a framework that generalizes speculative decoding to use multiple models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback.Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate.We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth.We validate PipeSpec across text summarization, mathematical reasoning, and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multidevice systems.Our code is available at https: //github.com/BradMcDanel/PipeSpec.