Hardware-Aware Pruning for Efficient Inference on Embedded Devices

With more powerful yet efficient embedded devices and accelerators being available for Deep Neural Networks (DNNs), machine learning is becoming an integral part of edge computing. In application scenarios with real-time requirements and multiple sensors connected to a single embedded device, such as autonomous driving, the optimization of DNNs plays a critical role during development and deployment to meet latency targets. However, optimization methods like pruning do not consider the underlying hardware platform. In this work, we propose a hardware-aware extension for existing pruning algorithms to optimally utilize the target hardware platform without requiring direct access to the device. We use estimation models to find Parto-optimal points as pruning targets and apply three different optimization strategies to optimize latency (Stacking) or accuracy (Clipping), or to find a trade-off between both (Rounding). Our experimental results on embedded devices like a Nvidia Jetson Nano or a low-power accelerator like the Hailo-8 show a latency decrease of 21.44% on average when optimization for latency (Stacking), an accuracy increase of 2.46% when optimizing for accuracy (Clipping), and a latency decrease of 7.46% together with an increase in accuracy of 1.32% for Rounding across multiple DNNs, datasets, target sparisities, and embedded devices when compared to standard pruning.