<scp>GraphPipe:</scp> Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism
Deep neural networks (DNNs) continue to grow rapidly in size, making them infeasible to train on a single device (e.g. GPU). Pipeline parallelism is commonly used in existing DNN systems to support large-scale DNN training by partitioning a DNN into multiple stages, which concurrently perform DNN computation for different micro-batches of training samples in a pipeline fashion. However, existing pipeline-parallel approaches only consider sequential pipeline stages and thus ignore the topology of a DNN, resulting in missed model-parallel opportunities.
Authors
- Colin Unger
- Xupeng Miao (ORCID: https://orcid.org/0000-0002-9371-8358)
- Tianqi Chen (ORCID: https://orcid.org/0000-0002-5744-3940)
- Daiyaan Arfeen (ORCID: https://orcid.org/0009-0009-5626-4551)
- Gregory R. Ganger (ORCID: https://orcid.org/0000-0002-3065-7316)
- Sunghyun Park (ORCID: https://orcid.org/0000-0003-4793-9069)
- Zhihao Jia (ORCID: https://orcid.org/0000-0002-1270-5185)
- Byungsoo Jeon (ORCID: https://orcid.org/0000-0002-6321-2989)
- Peiyuan Liao (ORCID: https://orcid.org/0000-0002-4387-7940)
- Sunghyun Kim (ORCID: https://orcid.org/0000-0001-8200-3106)
- Mohammad Alizadeh (ORCID: https://orcid.org/0000-0002-2002-2632)
- Shiyi Cao (ORCID: https://orcid.org/0009-0005-6834-375X)
- Mengdi Wu
- Neeraj Aggarwal
Institutions
- Berkeley College (US)
- Palo Alto University (US)
- Nvidia (United States) (US)
- Massachusetts Institute of Technology (US)
- Carnegie Mellon University (US)
- University of California, Berkeley (US)
- Stanford University (US)
Publication Details
- Published
- 2025-02-06
- DOI
- https://doi.org/10.1145/3669940.3707220
- Citations
- 5
- Primary Topic
- Brain Tumor Detection and Classification
- Type
- article
- Field-Weighted Citation Impact
- 4.21