HyperPipe: Bridging Heterogeneous GPUs for Accelerated Large-Scale Model Training

To enhance the performance of pre-trained foundation models in various natural language processing tasks, the scale of these models has been steadily increasing in recent years, driven by advancements high-performance computing hardware. However, the presence of diverse computing devices in data centers and the varying computational demands of different tasks make it challenging for a single type of high-performance G PU to suffice for the large-scale training of foundation models' and lower-performance GPUs often remain underutilized. Therefore, optimizing resource <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</sup>llocati<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">on</sup> and improving training efficiency have become a critical challenge. To address these issues, this study proposes a hybrid parallel training framework for heterogeneous computing devices, named HyperPipe. This framework automatically selects the optimal hybrid parallel strategy based on the performance characteristics of the devices, thereby achieving efficient resource utilization and acceleration training to meet the computational demands of large-scale FMs. Experimental results demonstrate that in a heterogeneous computing environment, training large models like GPT-3-1.3B with HyperPipe achieves significant acceleration, delivering speed-ups ranging from 1.6 x to 2.8 x compared to previous methods.