Speculative Inference with vLLM: Optimizing Heterogeneous Computing in Training-Inference Integrated Environments

Large Language Models (LLMs) have become central to advancements in AI, but their deployment in heterogeneous computing environments poses significant challenges in terms of performance and resource utilization. In this paper, we present a dual-model speculative inference framework based on the vLLM architecture that optimizes LLM inference between CPU-GPU systems. The framework uses dynamic collaboration, adaptive resource scheduling, and memory-aware techniques to address CPU resource underutilization, GPU inefficiency, and bottlenecks in highly concurrent scenarios. Extensive experiments show that our enhanced vLLM improves throughput by a factor of 86.3x in GPU-only settings and 56.3x enhancement in speculative inference compared to traditional Transformer-based systems. Speculative inference in vLLM improves performance by up to 70%, especially for small batches and long input-output sequences. We also investigate the impact of batch size, input length, and memory management on performance. The framework provides a scalable solution for optimizing LLM inference, with future work focusing on hardware scalability, adaptive scheduling, and extension to multilingual and multimodal models.