Bayesian Critique-Tune-based Reinforcement Learning With Adaptive Pressure for Multi-Intersection Traffic Signal Control

Adaptive traffic signal control (ATSC) is a critical component of intelligent transportation systems, with the capability to significantly alleviate urban traffic congestion. Reinforcement learning (RL)-based methods have demonstrated promising performance in achieving ATSC. However, due to the excessive trust of RL policies and the ineffectiveness of traffic movement representation, the existing methods are prone to making unreasonable policies. Such policies originate from instability in convergence or entrapment in local optima during training, and are reflected in poor decision-making performance. Therefore, this paper proposes a novel Bayesian Critique-Tune-based reinforcement learning with Adaptive Pressure (BCT-APLight) for multi-intersection signal control. In BCT-APLight, the Critique-Tune (CT) framework, a two-layer Bayesian structure is designed to refine the excessive trust of RL policies. Specifically, the Bayesian inference-based Critique Layer (CL) provides effective evaluations of the credibility of policies. When these evaluations are negative, the Bayesian decision-based Tune Layer (TL) fine-tunes the policies by minimizing the posterior risks. Meanwhile, an attention-based Adaptive Pressure (AP) mechanism is designed to effectively weight the effect of upstream lanes, thereby enhancing the rationality of traffic movement representation within the network. Extensive experiments conducted with a simulator over multi-intersection demonstrate that BCT-APLight is superior to other state-of-the-art (SOTA) methods on nine real-world datasets. Specifically, BCT-APLight decreases average waiting time (AWT) by <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><inline-formula><tex-math notation="LaTeX">$\boldsymbol{12.92\%}$</tex-math></inline-formula></b> over all tested datasets on average. Codes are open-sourced.