Failure-Aware Resource Allocation for Reliable Distributed Machine Learning Training
Long-running distributed training is exposed to worker failures and checkpoint overhead, yet lightweight GPU allocation rules often optimize failure-free throughput. This paper presents FARAS, a failure-aware resource allocation strategy that jointly selects checkpoint periods and allocates GPUs using estimated recoverable training progress. FARAS combines a per-worker failure hazard, nonlinear parallel speedup, checkpoint duration, and recovery delay in a marginal-goodput score. A minute-resolution Monte Carlo simulator evaluates five schedulers on synthetic batches of 14 training jobs sharing 24 GPUs, with 12 independently generated workload instances per condition. In the baseline failure setting, FARAS achieves a mean job completion time of 1,947.9 simulated minutes, compared with 2,037.8 for equal allocation with fixed checkpoints and 1,976.1 for size-aware allocation with adaptive checkpoints. These correspond to reductions of 4.41% and 1.42%, respectively. Across the tested failure intensities, FARAS consistently gives the lowest mean completion time among the examined policies. An additional paired experiment evaluates the impact of twofold underestimation and overestimation of worker failure hazards. The reported results describe synthetic system behavior rather than hardware measurements.
Authors
- Eva van Dijk
- Matthijs de Vries
Institutions
- University of Amsterdam (NL)
Publication Details
- Journal
- Academic Journal of Applied Sciences
- Published
- 2026-11-16
- DOI
- https://doi.org/10.54097/xqj1gs27
- Primary Topic
- Cloud Computing and Resource Management
- Type
- article
- Field-Weighted Citation Impact
- 0.00