Qasync: A Dual Network Architecture for Value Based Multi-Agent Reinforcement Learning

In this work, we propose Qasync, a novel dual network framework for value-based multi-agent reinforcement learning, aimed at addressing the control decision-making issues of Connected Autonomous Vehicles (CAVs) in merge scenario. The core contributions of Qasync include the following two aspects: 1) Qasync extends QMIX by integrating a feedforward network, which reduces the limitations of QMIX's linearity assumption. 2) Qasync introduces a periodic mixing mechanism that combines Q-values from both networks, enabling Qasync to better capture non-linear relationships and enhance agents' exploration of diverse strategies. We evaluated Qasync using the SMARTS platform in a mixed traffic merging scenario, where autonomous vehicles must merge onto a two-lane road from a slip road, interacting with vehicles that randomly generated at the starting point to simulate realistic traffic conditions. Qasync was compared against several value-based baselines, including VDN, QMIX, and QTRAN, achieving average merging success rates of 80%, 89%, and 94%, respectively. Qasync achieved an overall merging success rate of 95%, demonstrating its ability to effectively handle complex traffic interactions and highlighting its potential for enhancing autonomous driving systems.