TSM-LLM: Task Scheduling Management System for Large Language Models

As large model services gain popularity; the high cost of deployment presents a challenge. Thus, focusing on improving model efficiency in high-load scenarios while managing the scheduling of multiple models. This paper introduces TSMLLM, a Task Scheduling Management System designed for optimizing the deployment of large language models (LLMs). The proposed system leverages an adaptive time window to group realtime requests, optimizing the batching process for better resource utilization. A two-step scheduling method, involving filtering and scoring, is employed to tailor task distribution based on the specific characteristics of each large model. The system’s scalable architecture supports seamless expansion to accommodate evolving model requirements. Experimental results demonstrate a significant increase in throughput with only a slight rise in response time, contributing to a reduction in overall service costs.