What Should a Network-Aware Agent Observe? A Systematic Analysis of Metadata Features for Reinforcement Learning Control over Wireless Links

Recent work has shown that exposing Deep Reinforcement Learning (DRL) agents to network impairments during training allows them to perform better when deployed in realistic networks than ones trained under idealized conditions. We investigate systematically whether and what kind of information about the state of the network is beneficial to add to the agent’s observations. For this, we train PPO and SAC agents under a fixed impairment profile while varying the augmented observation space across eight feature groups on two common control tasks with different impairment sensitivity often seen in DRL research (CheetahRun and FingerTurnHard implemented in MuJoCo). Our results show that per-observation information about whether the packet was dropped or delayed recovers most of the performance of the trained algorithm, while adding statistics about latency, jitter, and packet loss provides little benefit or even destabilizes the training for PPO. SAC seems to be largely insensitive to the additional metadata. A window-size ablation shows that latency and jitter statistics provide no benefit across any tested window size, while the effects of the loss statistics are strongly window dependent. These results argue against a universal network metadata observation augmentation vector and in favor of algorithm- and task-specific feature selection.