Reverse Engineering System Dynamics from Reinforcement Learning Q-Functions in LQR Settings

This paper addresses the novel problem of recovering the dynamics of a linear system using data from reinforcement learning, particularly from the quality function associated with the Linear Quadratic Regulator (LQR). The quality function implicitly contains information about the system dynamics, but it is shown that it defines the system only up to a coordinate transformation that preserves the optimal value function. A characterization of this entire equivalence class of underlying dynamics consistent with a given quality function is given. The recovery of a unique physically meaningful model is resolved by framing the problem as an orthogonal Procrustes alignment, leveraging the same trajectory data used for reinforcement learning. Identifiability conditions and practical algorithms are provided, including a regularized variant for noisy data. A numerical example demonstrates that the method can exactly recover the true system dynamics in the noiseless case. This work provides an innovative link between model-free reinforcement learning and system identification, enabling the extraction of interpretable physical models from data-driven controllers.