PTMGS: A Cost-Optimal LLM Training Tasks Migration Method

Computing power infrastructure has a significant impact on the training speed and cost during the training process of Large Language Models (LLMs). Due to differences in GPU hardware, network architecture, and training frameworks, the cost and unit price of computing power fluctuate greatly, and even within the same cloud service provider, there are huge differences in prices for different clusters. The feature of regularly saving checkpoints during the training process of large language models makes it possible to dynamically migrate tasks between GPU clusters during the training process. Based on this background, this article proposes a set of greedy strategy heuristic algorithms with the goal of minimizing training costs, which choose the best time and target for task migration, thereby achieving overall optimization of training costs. Through multiple rounds of experimental verification on real task samples and simulation data, the results show that the proposed greedy strategy based preemptive algorithm can reduce the overall training cost by more than 10% . This study provides new solutions and methods for efficient computing power supply for large language model training, which can help reduce training costs for computing infrastructure operators and customers.