Causality-driven RL-based Scheduling Policies for Diverse Delay Constraints

We investigate the role of causal models in the context of obtaining scheduling policies that minimize delay violations. We consider multi-user queuing systems with random delay constraints, packet sizes and arrivals in Gilbert-Elliot wireless channels. Owing to Judea Pearl’s landmark work on causality to achieve a higher level of cognitive ability, we demonstrate the role of counterfactual reasoning, leading to the well-investigated optimal EDF policy for wired channels. Due to the randomness associated with the wireless channels, finding an optimal policy is not straightforward, leading to RL-based approaches. We present CPPO (counterfactual-PPO) and CA2C (counterfactual-A2C) algorithms that use counterfactual examples generated using causal models during the training process. We argue how stochastic gradient based policy gradient RL algorithms benefit during training due to incorporation of counterfactuals. We show that these algorithms provably lead to lower variance indicating a robust learning performance. Our results demonstrate a ~ 60% increase in the number of cases where CA2C and CPPO outperform their non-counterfactual counterparts with reduced variance and negligible computation overhead.