Fast On-device LLM Inference with NPUs
On-device inference for Large Language Models (LLMs), driven by increasing privacy concerns and advancements of mobile-sized models, has gained significant interest. However, even mobile-sized LLMs (e.g., Gemma-2B) encounter unacceptably high inference latency, often bottlenecked by the prefill stage in tasks like screen UI understanding.
Authors
- Ruiqi Liu (ORCID: https://orcid.org/0009-0004-3188-0976)
- Daliang Xu (ORCID: https://orcid.org/0000-0002-6775-0688)
- Hao Zhang
- Xuanzhe Liu (ORCID: https://orcid.org/0000-0002-7908-8484)
- Mengwei Xu (ORCID: https://orcid.org/0000-0001-6271-6993)
- Gang Huang (ORCID: https://orcid.org/0000-0002-4686-3181)
- Liming Yang (ORCID: https://orcid.org/0009-0009-1163-028X)
Institutions
- Beijing University of Posts and Telecommunications (CN)
- Peking University (CN)
Publication Details
- Published
- 2025-02-06
- DOI
- https://doi.org/10.1145/3669940.3707239
- Citations
- 15
- Primary Topic
- Advanced Data Storage Technologies
- Type
- article
- Field-Weighted Citation Impact
- 22.31