Enabling Parallelism Hot Switching for Efficient Training of Large Language Models
Training of large-scale deep learning models necessitates parallelizing the model and data across numerous devices, and the choice of parallelism strategy substantially depends on the training workloads such as memory consumption, computation cost, and communication cost. Current approaches generally assume uniform training workloads across samples in a given task. Thus, existing systems are designed to adopt a static parallelism strategy throughout one training process. Nevertheless, when training models with sequence inputs, this assumption fails due to the sequence length variation across samples. Consequently, training with a static parallelism strategy would result in sub-optimal performance.
Authors
- Xupeng Miao (ORCID: https://orcid.org/0000-0002-9371-8358)
- Haoyang Li (ORCID: https://orcid.org/0009-0001-5342-0194)
- Fangcheng Fu (ORCID: https://orcid.org/0000-0003-1658-0380)
- Xiaonan Nie (ORCID: https://orcid.org/0000-0001-6766-757X)
- Bin Cui (ORCID: https://orcid.org/0000-0003-1681-4677)
- Hailin Zhang (ORCID: https://orcid.org/0009-0000-4188-7742)
- Xuanyu Wang (ORCID: https://orcid.org/0009-0001-4811-5048)
- Y X Wang (ORCID: https://orcid.org/0009-0000-8375-493X)
- Houyang Ge (ORCID: https://orcid.org/0009-0001-3367-7486)
- Sheng Lin (ORCID: https://orcid.org/0009-0002-1495-0499)
Institutions
- Peking University (CN)
- Purdue University West Lafayette (US)
Publication Details
- Published
- 2024-11-04
- DOI
- https://doi.org/10.1145/3694715.3695969
- Citations
- 4
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 1.39
Funders
- Tencent
- National Natural Science Foundation of China
- China Postdoctoral Science Foundation
- Peking University
- National Science and Technology Major Project
- National Postdoctoral Program for Innovative Talents