Autonomic Resource Harvesting in HPC: Control Methods and their Reusability

High Performance Computing (HPC) systems are subject to dynamical variations occurring in e.g., jobs execution duration, I/O quantity, network consumption. Adapting to these unpredictable variations requires using autonomic management in an online feedback loop. The introduction of control theory methods allows for the design of well-founded autonomic managers. Choosing the relevant approach is daunting due to the variety of existing controllers. The criteria are of different natures, involving performance and efficiency, but also required expertise in control theory, and reusability or portability between sub-systems. Therefore, there is a need for comparative studies to assist designers choices. We consider the problem of resource harvesting in HPC systems, where scheduling often leaves resources idle. Our approach controls – through a feedback loop – the injection of small jobs in order to maximize the resources’ usage. The control problem is to manage the trade-off between harvesting and performance, in a reusable manner. We study how reusability relates to the adaptivity and robustness properties in control. We illustrate our approach with the classic Proportional-Integral-Derivative (PID) control, its upgrade as adaptive control, and Model-Free Control (MFC). We target CiGri , a system harvesting idle resources in a computing grid. We perform experimental evaluation and compare performance and reusability. Trade-offs are found on different criteria: while adaptive control is largely portable, its design complexity is significant for non-experts; PID control has good nominal performance, yet its portability is limited; MFC requires few competences to be used, but cannot provide strong guarantees.