FlowLoader: an efficient large language model loader via parallel transmissions and local multi-tier caching

The checkpoint of large language models (LLMs) typically range from tens of gigabytes to terabytes in storage size, with loading processes often requiring several seconds to minutes. This substantial latency issue significantly impacts service level objective compliance for providers of large-scale model services. To address the LLM loading cold-start problem, we present LLM FlowLoader, a model loader featuring multi-tier local caching and parallel communication pipeline optimization. By strategically leveraging GPU servers’ heterogeneous network resources and storage hierarchies, our system accelerates checkpoint loading through two key innovations: (1) An asynchronous pipeline transporter based on task stealing and affinity domain awareness, which transforms complex network transmission into simplified chain-structured transmission by introducing affinity domains, and adaptively enhances I/O bandwidth through a task-stealing algorithm. (2) A multi-tier caching system co-designed with the transporter that exploits GPU servers’ multi-tier memory/storage architecture to accelerate data accessibility and reduce cold-start latency. Experimental results demonstrate that LLM FlowLoader not only achieves significant loading speed improvements but also provides novel insights for optimizing GPU resource utilization.