Multi-Agent Q-Learning and Max-Pressure Control for Large-Scale Traffic Signal Control

Traffic congestion is becoming an increasingly critical issue, requiring urgent and effective solutions. As a data-driven method, reinforcement learning (RL) enables large-scale adaptive traffic signal control (TSC) in transportation systems. The decentralized multi-agent reinforcement learning (MARL) framework proves effective in scenarios involving multiple intersections. However, there is a lack of an effective behavior policy for agents to explore actions in the TSC scenario. A novel behavior policy is proposed by introducing the max-pressure control method in the transportation field into the <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\epsilon$</tex>-greedy behavior policy. An agent selects an action with the maximum pressure phase to maximize intersection traffic throughput, enabling each agent to choose an appropriate action according to real-time intersection traffic conditions in the TSC scenario. A multi-agent Q-learning algorithm utilizes the novel behavior policy for large-scale TSC. Experimental results show that compared to other existing decentralized MARL algorithms, the proposed approach performs well on a large synthetic grid under dynamic simulated traffic peak conditions.