Efficient Training of Large Language Models on Legacy GPUs with HetSeq and PyTorch

This study investigates the effectiveness and feasibility of using parallel machines with GPUs of different capacities to train large language models (LLMs) as an alternative to costly cloud platforms. The study explores optimization techniques focused on data, models, budget, and systems, detailing the configuration of a multi-GPU architecture and its implementation with CUDA software, PyTorch, and the HetSeq library, adapted to maximize performance in heterogeneous systems, including the usage of legacy resources (e.g., older GPUs). In performance tests, three approaches are compared: homogeneous load distribution, heterogeneous distribution, and HetSeq usage. The HetSeq configuration yielded a substantial performance improvement, achieving an execution time of 129,759 seconds, significantly lower than the 153,175 seconds observed for the homogeneous configuration and 165,387 seconds for the heterogeneous configuration in the maximum setup tested, representing a reduction of 15% and 21%, respectively. These results highlight HetSeq's advantage in optimizing training time as more GPUs are added, outperforming traditional approaches that attempt to standardize hardware utilization, such as PyTorch. Further analysis using Amdahl's Law reveals that approximately 82% of the training process is parallelizable, underscoring that optimized heterogeneous architectures are both viable and economically advantageous in scenarios that do not require homogeneous hardware.