Bandwidth Allocation for Heterogeneous HPC Data Ingestion using Dynamic Auctions

Modern High-Performance Computing (HPC) centers are facing significant challenges in ingesting large and diverse data streams, leading to bandwidth bottlenecks and scientific delays. To address the shortcomings of traditional static allocation and simple queuing methods, this work introduces a dynamic, value-based approach to bandwidth allocation. We propose two new auction-based mechanisms: the computationally efficient Greedy Value Density Auction and the theoretically robust Vickrey-Clarke-Groves (VCG) Knapsack Auction. Both mechanisms utilize user-provided bids that specify data requirements and scientific value, with the goal of maximizing the total value of successful data transfers. Our simulation results, based on realistic workload characteristics, demonstrate that these auction mechanisms significantly outperform standard First-Come, First-Served (FCFS) baselines. In high-load scenarios, our methods reduce average and tail completion delays by over 80% and improve predictability by decreasing the coefficient of variation of delay by up to 85%. Furthermore, network stability is enhanced, with the peak-to-average load ratio dropping by as much as 70%. This value-driven, adaptive strategy alleviates congestion, improves bandwidth utilization, and ensures that access is prioritized based on scientific importance, ultimately accelerating scientific discovery.