Deep Reinforcement Learning by Enhancing TD3 with Multiple Q-critics and Different Aggregation Functions

Twin Delayed Deep Deterministic Policy Gradient (TD3) is a famous reinforcement learning algorithm which continues to generate state-of-the-art results since its introduction in 2018. In this article, we present a generalization of TD3, called Triple Delayed Deep Deterministic Policy Gradient (3D3), which introduces additional Q-critics and a flexible update function. We analyze its behavior for different specifications when applied to benchmark datasets and compare its performance to TD3 and another extension proposed in the literature. Moreover, we discuss its mathematical properties from a theoretical point of view, including estimation bias and convergence properties under discrete state-action space assumptions.