MEPipe: Democratizing LLM Training with Memory-Efficient Slice-Level Pipeline Scheduling on Cost-Effective Accelerators (Artifacts)

The training of large language models (LLMs) typically needs costly GPUs, such as NVIDIA A100 or H100. They possess substantial high-bandwidth on-chip memory and rapid interconnects like NVLinks.The exorbitant expenses associated with LLM training pose not just an economic challenge but also a eocietal one, as it inrestricts the ability to train LLMs from scratch to a select few organizations. There is a significant interest in democratizing access to LLM training. This paper explores a potential solution by employing innovative parallel strategies on more affordable accelerators. Budget-friendly options like NVIDIA RTX 4090, while considerably less expensive and comparable in computational power to A100, are hindered by their limited memory capacity and reduced interconnect bandwidth, making the effective training of LLMs challenging. Conventional parallel strategies often result in high communication costs or excessive memory usage. Our paper introduces MEPIPE, a novel approach that includes a slice-level scheduling method for sequence pipeline parallelism. This method minimizes memory consumption without incurring additional communication overhead. Besides, MEPIPE utilizes fine-grained weight gradient computation to reduce idle time and mitigate imbalanced computation among slices. MEPIPE has demonstrated up to $1.68\times$ speedup ($1.35\times$ on average) on clusters equipped with 64 NVIDIA 4090 GPUs when training Llama models of varying sizes. 35\% Model FLOPS Utilization (MFU) is achieved in training Llama 13B model, being $2.5\times$ more cost-effective than A100 clusters.