Learning Simultaneous and Sequential Decisions in Multi-Agent Systems With Application to Traffic Signal Control

Efficient traffic signal control (TSC) has been one of the most useful ways for reducing urban road congestion. By modeling each intersection as an autonomous agent, multiagent reinforcement learning (MARL) shows remarkable performance in solving dynamic TSC. However, most of TSC methods based on MARL suffer from a non-stationarity problem since agents update their policies simultaneously. To resolve this issue, this paper considers multi-intersection TSC as a multi-agent sequential decision-making process with policy online learning. We utilize a sequential model such as Transformer architecture to learn the multi-agent joint policy. By carefully designing the advantage function of each agent, the monotonic improvement property can be guaranteed. Moreover, to fully exploit the advantages of both simultaneous and sequential MARL, we further propose a novel MARL network selection algorithm (MARL-NS) which selectively employs simultaneous MARL only at states that sequential MARL might fall into local optimum. Our theory proves that MARL-NS preserves cooperative MARL converge properties. Finally, we validate the proposed MARL-NS method on a unified TSC benchmark, LibSignal. Experimental results show that our method can outperform the baseline methods in network-level and arterial coordination.