Improving Speculative Decoding with Dynamic Adjustment and Probability Smoothing

Large language models (LLMs) have shown excellent performance in natural language processing tasks, but their autoregressive text generation incurs significant computational overhead. Speculative decoding (SD) improves generation efficiency by generating candidate tokens from a draft model and validating them in parallel with the target model. However, the distributional difference between the draft and target models limits the acceptance rate of the tokens. This paper, based on the Multi-Candidate Speculative Decoding (MCSD) method, proposes two innovations: first, dynamically adjusting the probability distribution of the draft model's tokens using KL divergence to reduce the distribution gap with the target model; second, smoothing the mixed distribution into a normal distribution to further enhance generation stability. Experimental results demonstrate that, compared to the standard MCSD method, the proposed approach significantly improves the acceptance rate between multiple data sets. Ablation experiments also validate the effectiveness of the two innovations.