Towards Energy-Efficient Edge Inference in Radio Cpns: a Mixture-of-Depths Transformer Based Tri-Parallel Distributed Approach

Large language models (LLMs) have shown remarkable abilities by significantly scaling up model size, but this has also greatly increased their computing overhead. Traditional solutions to reduce the overhead are offloading inference tasks to cloud servers. With the evolution of computing power networks, computing resources are increasingly distributed at the network edge, allowing inference tasks to be handled locally. This edge inference can reduce traffic stress on backbone networks from cloud offloading and improve the utilization of heterogeneous edge computing power. However, the challenge is how to balance the gigantic computing workloads of LLMs with the limited computing power at the edge. To overcome this, a mixture-of-depths (MoD) Transformer based tri-parallel distributed approach was introduced. By dynamically allocating computing power to specific positions in the Transformer and parallel computing, this approach maximizes the capabilities of heterogeneous edge nodes to achieve resource-efficient inference. Simulation results showed that the proposal performs best in various edge environments, reducing inference delay by up to 24.3 % and energy consumption by up to 37.5%, respectively.