Distributed Traffic Signal Control Model for Accurate Policy Learning Under Dynamic Traffic Flow: A Graph Forecast-State Vector Driven Deep Reinforcement Learning Framework

The increasing severity of traffic congestion has driven the application of deep reinforcement learning (DRL) in traffic signal control. It utilizes spatial states to interact with environments and develop control policy, which has been extensively studied and demonstrated to be efficient. However, the periodic fluctuations in traffic flow may produce states with the same spatial state but different trends. Relying solely on spatial states for interactions will mix up interactive experiences of these states, and bring instability to control policy learning for dynamic traffic flows. Moreover, many studies concentrate on vehicles as the main focus of traffic signal control, overlooking the demands of pedestrians, who are also vital participants in traffic. Therefore, this study proposes a distributed DRL traffic signal control method Graph Forecast-Vector Dueling Deep Q Network (GF-VDDQN) based on state vector. This method combines graph neural networks prediction and deep reinforcement learning, with the key idea being construction of state vector for agents to understand environments and learn policy accurately. Considering the principle of ‘people-oriented’, a comprehensive reward with green duration loss and vehicle waiting time is designed. A continuous duration control action set is introduced to enhance intersection utilization. Additionally, the distributed GF-VDDQN allows agents to collaborate with each other. To evaluate performances of GF-VDDQN, we conduct experiments in the SUMO traffic simulator, comparing it with advanced and traditional traffic signal control methods. Results show that state vector can assist agents in accurately understanding environments and avoiding experience mixture problem. GF-VDDQN performs superior than other methods.