Research on Distributed Training Architecture for Large Scale Models for Natural Language Processing

Large-scale language models are critical for natural language processing, and efficient distributed learning is needed to train such models. In this paper, a distributed training scheme is developed to improve the efficiency and scalability of training large-scale language models. This training scheme is divided into a three-layer structure, and also incorporates an autotuned gradient compression method and a load balancing mechanism to improve training efficiency at any time. Experiments show that this scheme is highly efficient in terms of computation and resource utilization for large-scale training tasks, is much faster than current open systems, and offers new ideas for faster training of large-scale language models.