On the stability of Lipschitz continuous control problems and its applications to reinforcement learning

We address the crucial yet underexplored stability properties of the Hamilton–Jacobi–Bellman (HJB) equation in model-free reinforcement learning contexts with a focus on Lipschitz continuous optimal control problems. By an established connection between Lipschitz continuous optimal control problems and classical optimal control problems, we provide new insights into the stability of the value functions. Specifically, we investigate the generic convergence property as well as the rate of convergence of value functions as the Lipschitz constraint increases. Moreover, we present a generalized Hamilton–Jacobi based reinforcement learning framework that accommodates general \(L^{p}\) geometries in the control regularization. To evaluate the stability properties and performance of our proposed method, we conduct various numerical experiments using benchmark examples and compare its efficacy with existing approaches.