Network-Centered Resource Management for HPC Networks

The significance of High-Performance Computing (HPC) cannot be overstated in today's research and industry landscape. Supercomputers play a crucial role in solving the most challenging problems in various domains, such as climate modelling, decoding the genome of cancerous cells, and training language models like ChatGPT. However, a significant problem with modern large-scale HPC systems is the underutilization of the available computing power, typically hovering around 3% for common HPC workloads. This not only results in the waste of money and energy but also limits innovation. The inefficiency is rooted in the communication demands of today's applications, causing bottlenecks in the HPC's interconnection network. This presents an opportunity to enhance system efficiency through network-oriented resource management. The goal of our research is to improve the effective computing power of supercomputers by developing new resource management strategies. We advocate that flexibility and direct control over resources are the fundamental principles for improving efficiency. State-of-the-art approaches concentrate solely on managing the compute resources and only consider static application and network behaviour. We plan to improve resource managers by introducing methods for real-time monitoring and modelling large-scale networks. Utilizing these methods, we aim to develop scalable strategies for managing both compute and network resources. Furthermore, we will explore the potential of elastic job scheduling. Through these efforts, we aim to overcome the network bottlenecks and prepare HPC systems for the future.