Time-Efficient Uncertainty Estimation Based on Target Networks in Deep Reinforcement Learning

Driven by the growing challenges of safety, exploration, generalization, and robustness in real-world applications, uncertainty estimation has emerged as a significant research domain in deep reinforcement learning (DRL). Uncertainty estimation theories can be broadly categorized into two main approaches: Bayesian theory-based methods and ensemble theory-based methods. However, Bayesian theory-based methods try to approximate the posterior distribution to estimate the uncertainties, and ensemble theory-based methods train multiple neural networks as a network ensemble to approximate the real distribution. They both cause a significant increase in training time. To address the above problems, we propose a target network-based approach for efficient uncertainty estimation (UEDQN), which uses time-updated target networks to replace the training of extra network. Specifically, UEDQN employs Bayesian theory to model the uncertainties, thereby distinguishing aleatoric uncertainty and epistemic uncertainty. Then, the two uncertainties are quantified and estimated using two periodically updated target networks. Finally, estimated uncertainties are used for exploration to accelerate the efficiency of training. The experimental results demonstrate the superior training efficiency of our method in environments with uncertainties and the excellent performance of the final trained agents.