Research on LLM speculative inference in training-inference integrated computing infrastructure scenarios

Recent advances in Large Language Models (LLMs) have highlighted critical challenges in inference cost optimization and resource utilization efficiency. Current AI computing centers face a significant imbalance in resource utilization, with CPU resources often underutilized (30-50% usage) while GPU resources remain costly. To address these challenges, this paper proposes a novel CPU-GPU collaborative speculative inference framework for LLMs. The framework achieves efficient resource utilization by deploying Small Language Models (SLMs) on CPUs working in conjunction with LLMs on GPUs. We implemented and validated our system on a platform equipped with 4th Generation Intel Xeon processors and H800 GPU, focusing on key performance parameters including batch size, input length, and output length. Our comprehensive experiments with different model combinations (68M/160M SLM paired with 7B LLM) demonstrate significant improvements in system throughput and effective CPU resource utilization while maintaining inference quality. The results show up to 1.25× speedup in throughput and increased CPU utilization from 30% to over 70%. Through extensive performance analysis across various configurations, we identified optimal operational parameters and resource allocation strategies. This research provides a practical solution for LLM inference optimization and offers valuable insights for improving resource utilization efficiency in AI computing infrastructure.