MaRVIn: A Cross-Layer Mixed-Precision RISC-V Framework for DNN Inference, from ISA Extension to Hardware Acceleration

The evolution of quantization and mixed-precision techniques has unlocked new possibilities for enhancing the speed and energy efficiency of Neural Networks (NNs). Several recent studies indicate that adapting precision levels across different parameters can maintain accuracy comparable to full-precision models while significantly reducing computational demands. However, existing embedded microprocessors lack sufficient architectural support for efficiently executing mixed-precision NNs, both in terms of ISA extensions and hardware design. This limitation results in inefficiencies such as excessive data packing/unpacking and underutilized arithmetic units, leading to performance bottlenecks. In this work, to address these challenges, we propose novel ISA extensions and the micro-architecture implementation specifically designed to optimize mixed-precision execution, enabling energy-efficient deep learning inference on RISC-V architectures. We introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MaRVIn</i>, a cross-layer hardware-software co-design framework that enhances power efficiency and performance through a combination of hardware improvements, mixed-precision quantization, ISA-level optimizations, and cycle-accurate emulation. At the hardware level, we enhance the ALU with configurable mixed-precision arithmetic (2-, 4-, and 8-bit) for weights and/or activations. To further improve execution efficiency, we employ multi-pumping to reduce execution latency and implement soft SIMD for efficient 2-bit operations. We also extend ISA to support these mixed-precision operations. At the software level, we integrate a pruning-aware fine-tuning method to optimize model compression. Additionally, we introduce a greedy-based design space exploration (DSE) approach to efficiently search for Pareto-optimal mixed-quantized models. Finally, we incorporate voltage scaling to boost the power efficiency of our system. Our extensive experimental evaluation over widely used DNNs and datasets, such as CIFAR10 and ImageNet, demonstrates that our framework can achieve, on average, 17.6× speedup for less than 1% accuracy loss and outperforms the ISA-agnostic state-of-the-art RISC-V cores, delivering up to 1.8 TOPs/W.