Multi-Agent Reinforcement Learning-Based Channel Access Optimization for IEEE 802.11bn

The up-to-date project authorization request (PAR) for IEEE 802.11bn envisions achieving Ultra High Reliability capability on the basis of the Extremely High Throughput Wi-Fi (IEEE 802.11be). Specifically, it calls for optimizing the 95th percentile of the latency distribution and MAC Protocol Data Unit (MPDU) loss while ensuring high throughput. The vision is challenging due to the competing nature of Wi-Fi channel access, especially in the case of overlapping basic service set (OBSS). This challenge gives rise to an emerging research topic of Wi-Fi, i.e., low-latency channel access. In this paper, we materialize low-latency channel access via multi-agent reinforcement learning (MARL). To meet the Wi-Fi legacy requirement, we retain the carrier sense multiple access with collision avoidance (CSMA/CA) protocol but enhance it by intelligently adjusting the channel access parameters such as the contention window (CW) and clear channel assessment (CCA) threshold. We model the OBSS scenario as multi-agent decision making and propose a distributed MARL method with knowledge transfer to optimize CW or CCA. This method leverages inter-agent communication to solve the locally observable problem. A corresponding fusion network is designed to minimize communication overhead. Innovatively, we introduce a relative reward to address the difficulty of finding labels. To further enhance the latency performance, we extend a joint optimization of CW and CCA threshold using hard parameter sharing network. Extensive simulation results show that the proposed method achieves an average gain of approximately 30% on CW optimization, 13% on CCA threshold optimization, and can fairly coexist with CSMA/CA protocol. Additionally, the joint optimization method demonstrates good generalization and achieves over 50% higher returns on random topologies.