Hardware-Aware Optimization of Large Language Models for Enhanced Throughput on Arm CPUs

Large Language Models (LLMs) have recently demonstrated versatile capabilities, approaching artificial general intelligence (AGI). The increasing scale of model parameters necessitates high-performance computing power, posing a challenge for efficient deployment within hardware limitations. For mainstream edge AI operation, it mainly involves using Arm CPUs to perform LLM inference. Thus, this work concentrates on enhancing the throughput of the prefill and decode phase by proposing a novel comprehensive optimization framework. A mixed-precision quantization technique compresses non-critical weights to low precision for parallel processing, while keeping critical weights high for accuracy. Additionally, integrating the computational process in the attention mechanism reduces the computational load. The proposed work is verified by optimizing the Qwen 1.8B model on the YiTian Arm CPU platform. Compared to the original model, this optimization achieves excellent throughput performance with 3.62 times higher for the prefill phase and 13.38 times higher for the decode phase. The PIQA test maintains an accuracy rate of 72.52%.