Performance-Energy Characterization of ML Inference on Heterogeneous Edge AI Platforms

Edge computing enables lower latency and secure real-time processing for machine learning (ML) workloads. State-of-the-art edge platforms are heterogeneous systems with compute cores of diverse performance and energy capabilities. This diversity motivates the need to identify the strengths of individual compute cores for more informed heterogeneous implementations of workloads. In this study, we benchmark a state-of-the-art heterogeneous platform consisting of a multicore CPU, integrated GPU and NPU, on its performance on various ML workloads. We measure system performance in terms of latency, throughput, and Energy-Delay Product (EDP) to analyze the performance-energy tradeoffs of such systems across various configurations -independently on the CPU, GPU or NPU and a heterogeneous NPU-GPU configuration. Our results indicate the superiority of NPUs for energy-efficient inference of LLMs and CNNs. We find that the NPU offers up to 5.2× speedup over CPU for CNN inference. For LLM inference, the NPU achieves up to 4× speedup during the prefill stage, while the GPU offers a 22× speedup on the decode stage. We observe that hybrid NPU-GPU implementation of LLM inference is superior to CPU-only inference in terms of energy consumption and end-to-end (E2E) latency, with an EDP decrease of about 93%. We highlight the broader applicability of our results by discussing extensions to long-context inference in causal LLMs and emerging state-space models on heterogeneous platforms.