A Twin Primal-Dual DDPG Algorithm for Safety-Constrained Reinforcement Learning

This paper introduces a novel algorithm, Twin Primal-Dual DDPG (Twin PD3PG), for safety-constrained reinforcement learning (RL). The Twin PD3PG algorithm prevents unsafe behaviours during policy evaluation and training. Firstly, we employ the Augmented Lagrangian Method (ALM) to solve Constrained Markov Decision Process (CMDP) problems and dynamically adjust the penalty factors for constrained costs, thereby balancing the maximization of long-term rewards with the minimization of long-term constraint costs. Secondly, the problem of overestimating the value of rewards and long-term discounted constrained costs is mitigated, by constructing two critic networks with the same architecture to accurately estimate the action value function using a clipped Double Q-learning variant. Our evaluation of Twin PD3PG on a classic optimal control problem involving singular arcs demonstrates its superior accuracy and efficiency compared to previous methods.