Integrated Freeway Traffic Control Using <i>Q</i>-Learning With Adjacent Arterial Traffic Considerations

Numerous studies have shown the effectiveness of intelligent transportation system techniques such as variable speed limit (VSL), lane change (LC) control, and ramp metering (RM) in freeway traffic flow control. The integration of these techniques has the potential to further enhance the traffic operation efficiency of both freeway and adjacent arterial networks. In this regard, we propose a freeway traffic control (FTC) strategy that coordinates VSL, LC, RM actions using a Q-learning (QL) framework which takes into account arterial traffic characteristics. The signal timing and demands of adjacent arterial intersections are incorporated as state variables of the FTC agent. The FTC agent is initially trained offline using a single-section road network, and subsequently deployed online in a connected freeway and arterial simulation network for continuous learning. The arterial network is assumed to be regulated by a traffic-responsive signal control strategy based on a cycle length model. Microscopic simulations demonstrate that the fully-trained FTC agent provides significant reductions in freeway travel time and the number of stops in scenarios with traffic congestion. It clearly outperforms an uncoordinated FTC and a decentralized feedback control strategy. Even though the FTC agent does not control the arterial traffic signals, it leads to shorter average queue lengths at arterial intersections by taking into account the arterial traffic conditions in controlling freeway traffic. These results motivate a future research where the QL framework will also include the control of arterial traffic signals.