Enabling Interactive Education With Low‐Latency Large Language Models

ABSTRACT Large language models (LLMs) have transformed educational applications through personalized learning and intelligent tutoring systems. However, educational LLMs (EduLLMs) face significant deployment challenges due to their massive computational demands and autoregressive nature, particularly in resource‐constrained environments. This paper presents a novel approach for inference acceleration aimed at facilitating the deployment of EduLLMs on a commodity GPU. This technique substantially diminishes the memory footprint and the volume of data transfers between the CPU and GPU by strategically preloading critical neurons directly onto the GPU, thereby enabling rapid access. Concurrently, computations pertaining to non‐critical neurons are processed on the CPU. Implementation of our optimized approach, AccEduLLM, on a single NVIDIA RTX 3090 GPU using FP16 type, achieves 60.09 tokens/s, which is 8.32 faster than the previous work, while preserving the model's accuracy. The approach demonstrates particular effectiveness for variable‐length educational content like essays and textbooks.