Energy Forecasting in High Performance Computing Datacenters Using Machine Learning

The size and scope of large-scale high-performance computing (HPC) computer systems have continued to grow, especially with respect to power density and overall capacity as increasingly performant high-end general-purpose graphics processing units (GPGPUs) have paved the way for large-scale deep learning models across a range of application domains. Data centers and their operators must contend with increasingly constrained power caps at the same time as effectively managing overall energy consumption. Often, large data centers need to provide estimates of expected future demand and energy across a variety of timescales, including monthly, weekly, daily, and in some cases, even hourly forecasting. There can be significant financial incentives from the local power grid to accurately predict these energy needs. These incentives are often in the form of lower rates and reduced penalties for either over- and under-consumption. In this work, we make use of several strategies to make these predictions, ranging from simple bulk statistical models to more sophisticated machine learning (ML) techniques that leverage historical energy usage, past and present job characteristics, as well as dynamic system queue pressure to improve prediction accuracy. In this paper, we discuss these techniques, evaluate their costs, dependencies, assumptions and overall effectiveness using actual workload and energy data from a cluster of interest to the United States Department of Energy. We demonstrate that relatively accurate real-time datacenter energy predictions can be obtained by making use of a modest set of input training data and that these predictions can be refined to be even more accurate given additional information provided by a conservative backfilling scheduler.