Design of Extended RISC-V for Q-Learning Hardware Accelerator using HW/SW Co-Design

In this paper, we propose a novel RISC-V ISA extension for a reinforcement learning-based hardware accelerator called RISC-Q. We introduce new instructions i.e., q.store_constant, q.store, q.load, q.max, and q.update. Each instruction corresponds to a hardware module. The q.store_constant sets necessary constants for the Q-learning by storing it in dedicated registers. The q.store and q.load instructions write and read the data to the main memory. The q.max instruction controls the max seeker hardware module to find $\mathbf{Q}$-max values. The q.update instruction accesses the $\mathbf{Q}$-updater hardware module to update the $\mathbf{Q}$-value with the corresponding state, action, and reward. Experimental result shows that the RISC-V extended reinforcement learning hardware accelerator (RISC-Q), which is implemented in the Nexys FPGA board, outperforms the software implementation up to $1.5 \times$ speed up. The work is useful for edge-computing applications such as smart navigation, smart communications, and robotics.