All-in-one Approach for Large Language Models Inference

In this paper, we aim to address the large language models (LLMs) deployment challenges in the industry area that require efficient inference solutions to handle the computational demands. To address the challenge, we propose an all-in-one approach that integrates CPU hardware and software-optimized solutions for accelerating the inference performance of LLMs. This paper introduces two key hardware features, advanced matrix extension and high bandwidth memory. Their importance for LLM inference is also explained. Additionally, the software implementation of the proposed all-in-one approach is presented. The paper introduces multiple optimization strategies adopted by the all-in-one approach for LLM inference optimization. The experiment results show the token generation latency of the OPT model with 13B parameters only needs 30ms on the proposed all-in-one approach.