OffLight: An Offline Multi-Agent Reinforcement Learning Framework for Traffic Signal Control

Efficient traffic signal control (TSC) is crucial for reducing congestion and improving urban mobility. While multi-agent reinforcement learning (MARL) has shown promise in adaptive traffic management, its reliance on real-time interactions makes online training costly and impractical. Offline MARL, which learns from historical data, offers a safer and more scalable alternative but struggles with heterogeneous behavior policies in real-world datasets. We introduce OffLight, a novel offline MARL framework designed to handle policy heterogeneity in traffic signal control datasets. OffLight integrates importance sampling (IS) to correct for distributional shifts and return-based prioritized sampling (RBPS) to emphasize high-quality experiences. To model diverse behavior policies, it leverages a Gaussian mixture model variational graph autoencoder (GMM-VGAE) which captures spatial and temporal traffic dynamics and improves policy estimation. We evaluate OffLight across real-world urban traffic scenarios varying from small to large traffic signal networks. Results show that OffLight outperforms existing offline RL methods by reducing average travel time by up to 7.8% and queue length by 11.2% compared to state-of-the-art approaches. Unlike prior offline RL methods, OffLight effectively adapts to complex and mixed-policy datasets, making it more reliable for real-world deployment without risky online training. These results highlight OffLight’s potential to improve urban traffic management at scale. Our implementation is available at https://github.com/rbokade/offlight.