Deploy-Efficient and Fast Network Probing with Time-Series Foundation Models

Active probing, the practice of running data transfers for a short duration to collect or observe network metrics, is widely used for monitoring network performance, detecting anomalies, and optimizing data transfer algorithms. However, accurately estimating end-to-end throughput remains challenging due to the risks of self-inflicted network congestion during extended probing and the difficulty of balancing short probing durations with measurement precision. State-of-the-art transfer algorithms designed for large-scale HPC or Cloud data transfers rely on very short yet accurate periodic probing to achieve faster convergence to optimal solutions. To address these challenges, various approaches ranging from classical time-series forecasting to deep learning have been proposed to estimate metrics based on the first few seconds of data. While deep learning-based solutions outperform traditional time-series models, they often fail to generalize across diverse network environments with varying protocols, resources, and congestion levels, requiring significant data collection and retraining for each network. Furthermore, the evolving nature of networks renders these models obsolete within a short time. Consequently, running transfers for fixed durations and averaging metrics remains the most used approach despite its poor accuracy. To overcome these issues, we introduce npLLM, a scalable framework that leverages time-series foundation models for throughput estimation across dynamic, heterogeneous networks. By fine-tuning the pre-trained IBM Granite Time Series Foundation Model (TSFM) using data from local and production clusters, combined with data augmentation techniques, npLLM achieves high accuracy with very few samples in training, minimizing the need for extensive network-specific data collection. This transfer learning approach using the base foundation model enables efficient adaptation to new networks with minimal effort, significantly reducing setup time and resource requirements. Experimental results demonstrate that npLLM reduces probing error rate up to 70% compared to state-of-the-art solutions using as little as one-tenth of training data.