A 209TOPS/W Reinforcement Learning Processor with Full Speculation Exploitation and Inference-Training Parallel Processing

Reinforcement learning (RL) has shown remarkable performance in diverse domains. With the introduction of neural networks to RL, their ability to solve sequential decision problems without labelled data enables various applications, such as automatic navigation, au-tonomous control, and industrial automation. RL includes two tasks: experience collection and network update, as shown in Fig. 1. For experience collection, single-input inference with forward propagation is performed to obtain states and rewards, and the actions are taken accordingly. For network update, training through forward prop-agation, error propagation, and gradient calculation with batched in-puts is performed to update the network's parameters. Inference is performed on single-batched input from the environment, and training is performed on multi-batched inputs from the collected experiences. Speculation can be leveraged to reduce the computational complexity, but has not been applied to previous RL processors such as [1], [2]. Sequential execution of experience collection and network update is adopted in [1], [2], which limits the overall system latency. To address these issues, this work presents an energy-efficient RL processor that exploits full speculation with parallel processing in in-ference and training, achieving 7.1× and 7.3x higher energy and area efficiency, respectively with 89% lower latency than prior arts.