Empowering Dragonfly: A Lightweight and Scalable Distribution System for Large Models With High Concurrency

Artificial Intelligence Generated Content (AIGC) models typically have hundreds of billions of parameters, and developers experience prohibitively long pulling time from a central model registry to their local environments. Peer-to-peer (P2P)-enabled model distribution that pulls models from local peers within a cluster instead of the central model registry is emerging as a promising technique to reduce model pulling time. Nonetheless, such model distribution systems have to handle bursty concurrent pulling tasks. This may occupy the network bandwidth of some peers for a long time, thereby making the peers unable to respond to further pulling tasks. We thus aim to design a lightweight and scalable model distribution system to balance the network resource usage of peers, by proposing learning-driven algorithms to accurately predict network status between peers and implementing the design in real production environments. Specifically, we first propose a lightweight network measurement mechanism that combines active delay probing and passive bandwidth inference with low resource overhead. We also propose a learning-driven task scheduling algorithm based on a structural graph representation with a varied-multi-hop attention mechanism, to predict bursty patterns of concurrent pulling tasks. We then design an asynchronous model training and inference method to enable seamless incremental learning based on the dynamic network status data. We finally implement our system design and the learning-driven algorithm in a Cloud Native Computing Foundation (CNCF) project <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Dragonfly</monospace> that has already been publicly released since its version <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$v2.1.0$</tex-math> </inline-formula>. Real experiments in the Ant Group’s production environment show that our system reduces the total completion time by at least 10% and increases the average bandwidth utilization of peers by 20%, compared with mainstream systems and algorithms.