Stress-Testing USB Accelerators for Efficient Edge Inference

Several manufacturers sell specialized USB devices for accelerating machine learning (ML) on the edge. While being generally promoted as a versatile solution for more efficient edge inference with deep learning models, extensive practical insights on their usability and performance are hard to find. In order to make ML deployment on the edge more sustainable, our work investigates how operational and resource-efficient these USB accelerators really are. For that, we first introduce a novel and theoretically motivated methodology. It allows for comparing intricate model performance in terms of quality and resource consumption across different execution environments. We then put it into practice by studying the usability and efficiency of Google's Coral edge tensor processing unit (TPU) and Intel's neural compute stick 2 (NCS). In total, we benchmark over 30 models across nine hardware configurations, which reveals significant trade-offs. Our work demonstrates that USB accelerators are indeed capable of reducing the energy consumption by a factor up to ten, however this improvement cannot be observed for all configurations - more than 50% of the investigated models cannot be run on accelerator hardware and in several other cases, the power draw is only marginally improved. Our experiments show that the Intel NCS improves efficiency in a more stable way, while the TPU shows further benefits in specific cases but performs less predictably. We conclude that the tested USB accelerators are only beneficial for deploying standardized deep learning models more efficiently and cannot live up to the claims of being standardized out-of-the-box solutions.