Multi-Agent Deep Reinforcement Learning for Vehicular Resource Allocation: A Comparison Study of Different Agent Cooperation Levels

The spectrum sharing design (such as the underlay sidelink configuration in cellular networks) for vehicle-to-vehicle (V2V) and vehicle-to-network (V2N) links can improve the efficiency of spectrum utilization for vehicular networks. However, differentiated service requirements for V2V and V2N links and the greater complexity of interference management make it challenging for conventional centralized learning-based methods in the face of real-world deployment. This motivates the exploration of multi-agent deep reinforcement learning (DRL) solutions, while existing research has predominantly focused on cooperative setting that causes large communication overheads for real-world deployment. This paper proposed a decentralized resource allocation scheme for joint spectrum sub-band and V2V transmission power selection that utilizes federated learning (FL) to formulate a mixed setup to avoid frequent communication and enable easy model sharing, termed FL-MX for the design of FL and mixed DRL agent cooperation level. Different levels of cooperation among agents are also explored by implementing different state-of-the-art (SOTA) DRL-based resource allocation algorithms on the developed standard-compliant simulation platform. Through extensive simulations, it is observed that the proposed FL-MX manages to converge effectively and can match the best SOTA performance, gaining a 22.55% and 9.5% improvement in the V2V payload delivery ratio and V2N throughput compared to the random baseline. Such results make it promising for future research to investigate multi-agent DRL designs at a mixed agent cooperation level for high-performing network decision solutions with less frequent communication demands.