On the Convergence of Gradient Descent in Scalar Two-Agent Infinite-Horizon LQ Games

In the context of infinite-horizon general-sum linear quadratic (LQ) games, the convergence of gradient descent remains a significant yet not completely understood issue. While the convergence in the finite-horizon setting has already been proved in Hambly21, the extension to infinite setting is challenging. We focus on a specific instance, the two-agent scalar game, and prove the algorithm’s convergence in this simplified scenario. Using this example as a foundation, we demonstrate that even in the presence of equilibria that are saddle point of the gradient descent dynamics, gradient descent can still converge, and that the inclusion of noise is unnecessary.