MCCL: Measurement-based Collective Communication Library
Collective communication is a critical performance bottleneck in large-scale distributed training. Existing libraries and synthesis frameworks compute open-loop schedules once, assuming a static topology and peak link capacities. In production, however, persistent gray failures can silently degrade link throughput and latency for hours, causing pre-computed schedules to lose much of their efficiency during execution. In this paper, we present Measurement-based Collective Communication Library (MCCL), a collective communication library that closes the loop between runtime network measurement and dynamic schedule synthesis. MCCL passively measures GPU-to-GPU throughput, detects persistent degradation, re-synthesizes the affected communication-intensive schedules with warm-started synthesizers, and hot-swaps them without interrupting training. We validate the complete loop on a multi-node hardware prototype and use simulations to evaluate cluster-scale scenarios. In our experiments, MCCL incurs negligible overhead (<2%) while maintaining high measurement accuracy (<1.2% error) and low schedule hot-swap time (≈ 12 ms). Notably, under gray failures, MCCL reduces training time by up to 4.99x over static schedules on simulated training workloads for state-of-the-art LLMs such as Llama and DeepSeek.
Authors
- Massimo Gallo (ORCID: https://orcid.org/0000-0001-8781-0775)
- Gabriele Castellano (ORCID: https://orcid.org/0000-0002-1889-7675)
- Roberto Morabito (ORCID: https://orcid.org/0000-0002-4240-9934)
- Paolo Medagliani (ORCID: https://orcid.org/0000-0001-8520-3664)
- Sébastien Martin (ORCID: https://orcid.org/0000-0002-8528-6698)
- Michele Ferrero
Institutions
- EURECOM (FR)
- Laboratoire de Conception, Optimisation et Modélisation des Systèmes (FR)
- Université de Lorraine (FR)
Publication Details
- Journal
- Proceedings of the ACM on Networking
- Published
- 2026-11-19
- DOI
- https://doi.org/10.1145/3856969
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00