Infinite-horizon optimal control of nonlinear discrete-time systems: HJB pde, Hamiltonian dynamics and invariant manifolds

Nonlinear discrete-time optimal control problems are studied over an infinite horizon with the aim of establishing a connection between the solution of the Bellman equation and the trajectories of the Hamiltonian difference dynamics associated with the problem. First, a discrete-time counterpart of the Hamilton–Jacobi–Bellman partial differential equation is introduced and discussed. The latter is then further instrumental for showing that a certain manifold, involving the costate variable and the gradient of the value function, is invariant for the Hamiltonian dynamics, hence recovering a well-known property of continuous-time optimal control problems. This feature is then leveraged to envision an episodic learning strategy based on the notion of invariant manifold.