Training strategies, computational consumption, and memory control for large-scale language models

Large language models (LLMs) like ChatGPT show surprising results in many areas. In this context, more and more people focus on developing specific-domain-targeted LLMs and related LLM technologies. In this study, we focus on several important LLM training issues. Specifically, we focus on neural network training, optimizer types, and memory consumption. For memory consumption, we put forward some calculating techniques like Mixed Precision Training and Zero1-3. They are used to reduce the memory usage. We also applied tools like DeepSpeed and FSDP for optimization. In this study, we put forward different strategies for memory optimization. This is also to promote efficient and economic LLM construction and related techniques.