Impact of Reward Function Selection on DQN-Based Traffic Signal Control

As the scale of traffic increases with growing urban populations, traditional fixed traffic signal timing methods become less effective in regulating traffic at intersections. Reinforcement learning (RL) methods can dynamically optimize traffic signal timing and lead to alleviated traffic conditions and have become a popular approach recently. This study aims to explore the impact of reward function selection in these RL approaches and the relationship between these selections. To conduct this study, three reward functions for a Deep Q-Network (DQN) were trained on a benchmark intersection: average waiting time, vehicle speed, and CO<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> emissions. From these simulations, we found that the DQN algorithm was able to reduce the mean waiting time by 82.63%, increase the mean speed by 146.84%, and decrease the mean CO<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> emissions by 4.26%, respectively, compared to a fixed-timing traffic signal controller. We also found that the RL-based model has a level of robustness over a traffic scale that deviates from training conditions. In addition to the impact of the reward function on the designated evaluation metrics, we also studied the impact on other metrics. We found that (1) the optimization of the average waiting time was linked to an increase in speed and a decrease in CO<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf>emissions; (2) the optimization of the average speed was discovered to reduce waiting time and increase CO<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> emissions, and (3) the emission-targeted optimization produced an increased waiting time and decreased mean speed. These findings set expectations for DQN method performance for traffic signal control and indicate guidelines for reward function selection for multi-objective traffic control.