iPipe: An Efficient Adaptive Pipeline Parallel DNN Training System

Pipeline parallelism is an efficient approach for distributed deep neural network (DNN) model training. However, existing pipeline parallel training systems often employ fixed model allocation strategies, leading to suboptimal efficiency in dynamic training scenarios. To address this issue, this paper proposes an efficient adaptive pipeline training system, iPipe, which dynamically manages the stages of the DNN model on GPUs based on their real-time states, effectively enhancing the training efficiency. In iPipe, a model allocation algorithm based on dynamic programming method is designed to search for an optimal model allocation plan with low complexity. Additionally, a live migration strategy which adopts an alternating computation method is proposed to achieve dynamic migration of model parameters between workers with minimal data transfer. We evaluate iPipe on a real GPU cluster, and the experimental results demonstrate that iPipe outperforms other pipeline parallel training systems in terms of overall training time for DNN models.