FPGA-based Acceleration of Deep Q-Networks with STANN-RL

Deep Reinforcement Learning is a promising research domain with many interesting applications in various fields, from protein folding to real-time decision making in Internet of Things applications. The state-of-the-art algorithms require a large amount of computing resources to train and could benefit significantly from hardware acceleration. This paper presents a new hardware accelerator for the Deep Q-Network algorithm. The algorithm is implemented on a Xilinx Versal and integrated into a heterogeneous training system with a host PC that runs common reinforcement learning environments for benchmarking. The Deep Q-Network accelerator is highly configurable and achieves a $2.5 \times$ speed-up compared to a CPU baseline. The implementation uses the STANN library for the design of the neural network training and introduces STANN-RL, extending STANN with common reinforcement learning functionality. This facilitates the implementation of additional accelerators for Deep Reinforcement Learning algorithms in the future.