Distributed Inference Optimization for Large Language Model in Edge-Cloud Collaborative Networks

With the progressive evolution of large language models (LLMs) and the increasing need of computing for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{6 G}$</tex>, it becomes crucial for multiple network nodes with limited computing resources to share the need for large model inference. Model partition methods have been proposed to enable computation-intensive artificial intelligence (AI) services by splitting an AI model across multi-edge and cloud nodes. In this paper, a distributed inference optimization framework for transformer decoder-only based LLMs (DIO-LLMs) is proposed in edge-cloud collaborative networks. The partitioning and offloading strategy is determined based on the computing workload and network status. DIO-LLMs specifically accounts for the parallel execution capabilities of the transformer architecture. It employs a two-phase model partitioning strategy, comprising inter-layer and intra-layer partitions, to effectively distribute LLMs across edge and cloud nodes. Additionally, to mitigate inference latency under resource limitations, a Greedy Proximal Policy Optimization (GPPO) based algorithm has been developed to devise optimal strategies. Simulation results indicate that under memory constraints, the proposed algorithm can reduce inference latency more effectively than other baseline algorithms.