Swarm-Level Task Learning via Generalized Moments in Reinforcement Learning With Reward Machines

Swarm-level task learning provides a basis for learning complex tasks in swarm systems using swarm-level features. We propose a method, SwaRM-L, to learn reward machines (RMs) that encode non-Markovian reward functions. We use reward machines to specify the task and its temporal structure. Our approach enables a swarm of agents to learn an RM that eventually becomes equivalent to ground truth RM (i.e., the specified task) in this environment (agents have no access to the ground truth RM and only learn it through environment interaction). We use generalized moments (GMs) to characterize swarm features and estimate the RM state. Each agent maintains an estimated GM to contribute to the collective learning process. The agents use a gossip algorithm to communicate with neighbors and update their estimated GMs. We prove that our method converges to an optimal policy and learns an equivalent RM to the ground truth RM within the environment. We evaluate our proposed method in three case studies involving forty agents with homogeneous dynamics. Our results demonstrate the effectiveness of our method in learning complex swarm behaviors.