Model-Aided Deep Reinforcement Learning for Fast RAW Parameter Adaptation in Wi-Fi Halow Networks

In this paper, we consider a large-scale Wi-Fi HaLow heterogeneous network, where numerous stations (STAs) are distributed around an access point (AP) to collect and transmit data using the restricted access window (RAW) mechanism. The AP manages channel access by adjusting and broadcasting RAW Parameter Set (RPS) and grouping messages, which include the duration and slot allocation for each RAW group. We aim to maximize the overall network throughput while ensuring fairness among STAs. Traditional methods struggle with real-time RAW optimization in practical networks. To overcome this challenge, we first divide the RAW groups heuristically according to the STAs' task type and formulate the throughput optimization problem regarding various RPS. Then, we construct a virtual twin network environment to estimate the throughput performance used for RAW optimization. Specifically, the twin environment is built on neural networks and trained by both synthetic data generated from the classic Markov model and the NS-3 simulator. Given the throughput estimate, we devise the reward function of the proximal policy optimization (PPO) algorithm to adapt the optimal RPS, without frequent interaction with the real network environment. Numerical results indicate that the twin-enhanced PPO (TE-PPO) algorithm achieves a comparable performance with the classic PPO algorithm built on the real trace of the NS3 simulator. Particularly, TE-PPO can reduce the time overhead for RPS adaptation to 1/180.