Deep Reinforcement Learning Based on Spatiotemporal Information for Network-Wide Traffic Signal Coordination Control

Graph neural network-based deep reinforcement learning (GNN-DRL) algorithms have been widely applied in traffic signal coordination control of urban road networks. However, current traffic signal control methods based on GNN-DRL focus on integrating historical and current spatiotemporal data to represent features of traffic networks, without utilizing predicted future information to enhance traffic efficiency. This work proposes a novel spatiotemporal information-based deep reinforcement learning method for traffic signal coordination control of urban road networks. It implements a heterogeneous subgraph representation method to model spatial structures among closely related intersections, strengthening subgraph feature representations while reducing computational complexity. Additionally, a multi-scale spatiotemporal heterogeneous graph feature aggregation technique is designed. The proposed method incorporates traffic signal timing scheme, vehicle states and road network topology as graph node features. By applying a graph neural network, it captures multistep spatiotemporal information from historical, current, and predicted data, thereby enhancing network feature representation and foresight. Furthermore, a novel reward function is designed to perceive the spatiotemporal information of a road network. The function uses the betweenness centrality to evaluate the spatial importance of intersections. It introduces total number of vehicles and predicted traffic flow to dynamically assess the current traffic state and future traffic demand in the lanes. It improves the agent’s ability to perceive and use spatiotemporal information to make decisions. We evaluated our proposed method through experiments under three different traffic scenarios: low, medium, and high flows. The results clearly demonstrate that the proposed method outperforms existing state-of-the-art methods, by reducing average queue length by 34.12%-59.45%, maximum queue length by 25.31%-47.83%, lane occupancy rates by 27.22%-51.56%, and vehicle count by 27.29%-51.92%. Meanwhile, experiments on computational overhead and real-road networks further confirm that SIDRL offers advantages in terms of low cost and high performance. This presents new technical insights for the real-time deployment and resource optimization of urban traffic signal control.