WOLT: Transparent Deployment of ML Workloads on Lightweight Many-Accelerator Architectures

Most available frameworks to develop machine learning applications target software deployment on general-purpose processors or GPUs. We propose Wolt, an end-to-end efficient solution to run TFLite application workloads on lightweight system-on-chip (SoC) architectures that feature many fixed-function hardware accelerators. WOltenables the execution of TFLite operations on pre-designed accelerators without requiring any modification of the application source code. As it establishes an interface between the high-level framework and the device drivers of the accelerators, WOLtincludes a resource manager that allows multi-tenant and conflict-free hardware acceleration of many TFLite applications running in parallel on the SoC. We evaluated WOltwith a comprehensive set of FPGA-based experiments by profiling and running 13 different TFLite workloads on a variety of complete SoC prototypes. We designed these prototypes by combining many CVA6 RISC-V processors with multiple accelerators for vector-matrix multiplication and two-dimensional convolution. For these workloads, Woltdelivers up to 23.2 x performance speedup and up to 14.6 x energy-efficiency gains compared to a purely software execution. When running multiple TFLite workloads in parallel, Wolt achieves up to 4 x of additional performance gain compared to basic hardware acceleration, thanks to efficient resource management.