LLM Training Workload IO Characteristics

Deep learning models, especially large language models (LLMs), demand significant computational resources for training. To optimize performance and resource allocation, understanding memory utilization and I/O characteristics during LLM training is essential. In our study, we explore the impact of SSD storage technologies on the training of a BERT (Encoder-Based) model and extend our findings to other encoder and decoder-based models, including GPT-Falcon, RoBERTa, and GPT-2, on a GPU setup. We utilize regular checkpoints to analyzing and saving model state and parameters for accessing memory usage and assess write bandwidth (for checkpointing) and read bandwidth (for restore operation as well as regular traffic data ingestion) to identify bottlenecks. Additionally, we investigated the impact of different Logical Block Addressing (LBA) sizes on training characteristics. The synchronous checkpointing process was found to significantly impact training time, particularly with smaller sizes, due to frequent I/O operations. The contribution of this paper includes the systematic study of the different types of traffic to SSD and their characteristics during the LLM pre-training phase for a single GPU setup and the analysis of how this study is applicable for multi-GPU setup. This study offers valuable insights for both practitioners and researchers, and we hope that this study will lead to additional future work in improving the SSDs for the large-scale distributed training systems for LLMs.