Joint Caching and Inference for Large Language Models in Wireless Networks

To reduce the serving delay of large language model (LLM)-based applications, the edge-based LLM serving mechanism offers a promising solution by caching LLMs at the edge to provide LLM inference services closer to users. Motivated by this, we propose an edge-based LLM caching and inference framework to support low-delay LLM-based services. Based on the framework, we formulate a joint LLM caching, inference task scheduling, and computation resource allocation optimization problem to minimize LLM serving delay, where time-varying LLM popularity is considered. Given an LLM caching policy, we first obtain the optimal solution for the computation resource allocation and task scheduling by using traditional optimization methods. Then, we propose an improved double deep Q-network (IDDQN) algorithm that effectively learns the optimal LLM caching strategy under unknown LLM popularity. The IDDQN algorithm integrates a state coding and action aggregation (SCAA) mechanism in the deep neural network structure, enabling it to efficiently capture users' preferences for LLMs and mitigate the slow convergence issues due to the large action space. Simulation results indicate that the proposed scheme achieves both lower average user delay and faster convergence than other benchmarks.