EcoMLC: An Energy-Efficient Machine Learning Compilation Framework via Cache Reuse and Frequency Scaling

While modern machine learning compilers (MLCs) are pushing deep neural network (DNN) inference latency to its theoretical limits, the accompanying energy overhead has become a critical bottleneck for large-scale deployment. Excessive energy consumption not only increases electricity costs, but also affects the sustainability of edge platforms. Traditional performance-first optimizations often lead to significant energy waste, as they overlook the complex interplay between kernel configurations, cache behaviors and frequency scaling. This paper presents EcoMLC, an energy-efficient machine learning compilation framework. EcoMLC introduces a reverse scheduling policy combined with an explicit cache priority mechanism, mitigating cache thrashing inherent in standard least recently used (LRU) policies. EcoMLC employs an end-to-end frequency-sweeping analysis to identify optimal energy operating points and enables fine-grained trade-offs between performance and energy under diverse service level objective (SLO) constraints. Experimental results demonstrate that EcoMLC effectively optimizes various workloads, achieving an average of 43% and 26% energy savings (in J/request and J/token) over NVIDIA’s TensorRT and vLLM, and 22% energy savings on an AMD GPU compared to vLLM. For certain large language model (LLM) configurations, EcoMLC achieves a 33.9% reduction in energy consumption at the cost of a marginal 7% decline in performance.