Distributed Output Optimal Robust Time‐Varying Formation Tracking Control for Heterogeneous Multi‐Agent Systems Based on Reinforcement Learning

ABSTRACT In recent years, time‐varying formations have emerged as a crucial control strategy due to their distinct advantages in adapting to environmental changes. However, achieving time‐varying formation tracking in heterogeneous multi‐agent systems via reinforcement learning (RL), especially when the leader's state is unavailable, remains a significant challenge. This paper investigates the problem of time‐varying formation tracking for heterogeneous multi‐agent systems (MASs) with unknown dynamics and inaccessible leader states. A fully distributed output‐feedback reinforcement learning framework is developed, which integrates adaptive observer design, optimal regulation, and data‐driven policy iteration into a unified control scheme. Specifically, a fully distributed adaptive observer is proposed to estimate the leader's state using only its output, without requiring Laplacian eigenvalues or global network knowledge. Based on this observer, an output‐feedback reinforcement learning controller is constructed to achieve asymptotic convergence of the formation tracking error, in contrast to existing state‐feedback‐based methods that only ensure bounded errors. Furthermore, a state reconstruction mechanism, originally used in synchronization problems, is extended to time‐varying formation tracking, enabling policy learning directly from input‐output data under unknown dynamics. Theoretical analysis and simulation studies demonstrate that the proposed framework achieves robust, scalable, and model‐free time‐varying formation tracking, offering clear advantages over existing approaches.