Offloading with prefetch-aware memory optimization: enabling efficient single-GPU inference for large language models

Large Language Models (LLMs) have achieved significant progress in natural language processing, but their increasing scale imposes high memory and resource demands during deployment. This becomes particularly challenging for single-GPU setups with limited memory. To address this, we propose a novel memory optimization method, Offloading with Prefetch-Aware Memory Optimization, which combines traditional weight offloading with a lightweight prefetch-aware scheduling strategy. Our approach reduces GPU memory usage and improves inference throughput by asynchronously preloading weights, effectively mitigating the latency overhead from frequent CPU–GPU data transfers. Extensive experiments demonstrate that our method enhances deployment efficiency without compromising model accuracy, and it can be seamlessly applied to various Transformer-based models like LLaMA and OPT. The proposed solution is hardware-agnostic and model-agnostic, offering scalability and robustness for efficient deployment in resource-constrained environments.