QMLFQ: A Quantization-based Queue Scheduling Framework for Efficient LLM Serving

Fueled by advancements in Large Language Models (LLMs), applications like ChatGPT have ushered in a transformative era for AI-powered interactions. However, the computational demands of LLMs, characterized by their massive parameter size and intricate computations, pose a significant challenge to its model serving. Under high concurrency or resource-constrained scenarios, LLM service systems can experience substantial latency and potential crashes, jeopardizing user experience and system stability. To address this challenge, this paper presents a novel LLM serving framework that leverages dynamic queue scheduling and quantization switching to alleviate computational bottlenecks and reduce response latency. Specifically, we propose an enhanced scheduling algorithm based on the Multi-level Feedback Queue (MLFQ) that dynamically switches between various quantized-level models and prioritizes requests, thereby improving LLM inference parallelism and reducing job completion time (JCT) as well. Through extensive experiments conducted on open-source datasets, we demonstrate that our proposed approach effectively reduces JCTs compared to other leading baselines. This improvement in JCT ultimately leads to enhanced quality of service (QoS) for LLM-based applications.