Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression
Gradient aggregation has long been identified as a major bottleneck in today's large-scale distributed machine learning training systems. One promising solution to mitigate such bottlenecks is gradient compression, directly reducing communicated gradient data volume. However, in practice, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy.
Authors
- Ran Ben Basat (ORCID: https://orcid.org/0000-0003-0196-9190)
- Michael Mitzenmacher (ORCID: https://orcid.org/0000-0001-5430-5457)
- Shay Vargaftik (ORCID: https://orcid.org/0000-0002-0982-7894)
- Wenchen Han (ORCID: https://orcid.org/0009-0001-8262-5658)
- Brad Karp (ORCID: https://orcid.org/0000-0002-7587-5034)
Institutions
- Harvard University Press (US)
- University College London (GB)
Publication Details
- Published
- 2024-11-11
- DOI
- https://doi.org/10.1145/3696348.3696857
- Citations
- 6
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- article
- Field-Weighted Citation Impact
- 2.93