Model-free inverse reinforcement learning for linear discrete-time deterministic systems

Designing appropriate objective functions for autonomous agents is a challenging task that often involves cumbersome reward engineering. Inverse reinforcement learning (IRL) provides an alternative by inferring the underlying cost structure from observed behaviours. This paper introduces two novel model-free IRL techniques tailored to linear discrete-time deterministic systems. The first is an online method that leverages a single trajectory, enhanced with exploratory inputs, to quickly approximate the expert's objective. The second is an offline iterative approach that begins with a single dataset for parameter estimation and, if necessary, incorporates additional datasets to satisfy certain conditions, all while eliminating the need to repeatedly solve a forward control problem. Notable innovations of our work include a novel formulation of the discrete-time Hamilton-Jacobi-Bellman equation for online estimation and a new controller representation that facilitates model-free solution development in offline estimation. Numerical simulations validate the effectiveness of the proposed methods, demonstrating that the estimated cost functions produce learner policies that closely match those of the expert.