Leveraging RISC-V for HW/SW Codesign of Flexible and Efficient TinyML SoCs

High computing efficiency and flexibility are key for embedded artificial intelligence (AI) in extreme edge devices for Internet-of-Things (IoT) applications. Relying on heavily quantized fixed-point arithmetic is crucial for efficient computation on deep neural networks (DNNs), the leading AI models. However, classical microcontroller units (MCUs) lack instruction set architecture (ISA) support for low-bitwidth integer arithmetic, causing inefficiencies on inference for aggressively quantized DNNs (QNNs). In this work, we describe a set of ISA extensions and their efficient hardware implementation, leveraging the flexibility of the RV32 ISA to boost computing efficiency for QNN inference. We introduce low-cost custom RISC-V instructions and micro-architecture enabling flexible execution of 2-bit to 32-bit arithmetic and mixed-precision combinations. Additionally, we propose a custom execution paradigm for SIMD sum-of-dot-product operation, fusing a dot product with a load operation. Moving from single-core to multi-core MCUs with the goal of boosting performance without compromising energy efficiency, we show how to integrate ISA-enhanced cores in clusters of processors. We introduce a software-configurable vector lockstep execution mode (VLEM) to aggressively reduce instruction fetch and instruction cache access power for the cluster. To evaluate the proposed solutions in a homogeneous implementation fabric, we implement the cluster of processors in 22nm FDX technology node. Low-bitwidth mixed-precision MatMul kernels run up to 14× faster on the ISA-enhanced cores compared to the baseline RV32 cores, at a 12.5× higher energy efficiency. Compared to classic MIMD execution, the VLEM feature at cluster level reduces power by 35%, boosting the energy efficiency by up to 1.6×. With a peak performance of 73 GOPS and a peak energy efficiency of 5.32 TOPS/W, our approach demonstrates that an 8-core cluster can achieve energy efficiency levels comparable with dedicated accelerators, while retaining full flexibility.