Design Principles for Reinforcement Learning in Congestion Control Environments

Reinforcement Learning (RL) has emerged as a powerful tool for optimizing sequential decision-making processes. Recent years have seen growth in interest in applying RL to managing Congestion Control (CC) in computer networks, with the goal of handling diverse kinds of traffic that may be challenging for human-designed algorithms. However, implementing RL in CC is a complex process consisting of numerous design choices in developing the state space, action space and reward function design. This paper evaluates different design strategies for developing the state space and how they contribute to learning. We assess the impact of various action spaces on learning and investigate how the reward function influences network performance in different scenarios. Our findings reveal that metrics used in state space development significantly affect RL agent learning. We demonstrate that a discrete action space leads to an improvement in learning compared to a continuous action space. Through comparative analysis of 14 reward functions, we show that different reward designs result in varying trade-offs between throughput, delay, and loss. These insights enable the customization of RL agents to scenarios with low or high bandwidth.