Optimizing TinyEngine for the RISC-V Vector Extension

Embedded edge devices face strict energy and size constraints for IoT and WSN applications. Because of this, advances have been made to enable state-of-the-art edge neural network models, such as MobileNetv2, to decrease their peak memory requirements to tens to hundreds of kilobytes using network architecture search tools and highly optimized libraries. However, optimization work in this space has been limited to a specific set of microcontroller cores including the recently developed embedded RISC-V processors that can take advantage of the highly parallel nature of machine learning workloads. This work explores the optimization of the TinyEngine inference library for the T908 vector processor as a reference for embedded vector processor designs. The vectorization achieved a 2.83x speedup relative to a scalar portable version of TinyEngine running on the same core, without a corresponding fall in average accuracy. In addition, replacing floating-point quantization with fixed-point operations is shown to have only a minor 3-4% reduction in Top 5 accuracy with negligible impact on performance, indicating that integer-only vector implementations can also be viable for embedded machine learning workloads.