Bayesian Uncertainty Weighted Optimization for Offline Reinforcement Learning

Offline reinforcement learning (offline RL) endeavors to learn effective policies from a large batch of pre-collected datasets without any costly or dangerous online exploration. Nevertheless, offline RL always suffers from substantial algorithmic extrapolation errors and may fail when bootstrapping from out-of-distribution (OOD) actions or states. In this work, we introduce a practical and effective Bayesian uncertainty weighted optimization (BUWO) to leverage the Bayesian uncertainty to account for the epistemic uncertainty associated with each training sample and penalize the state-action pairs with high uncertainty. We compare BUWO with other prevailing offline RL algorithms on D4RL benchmarks. The experimental results demonstrate that the algorithm can enhance the average reward score by almost 15% without additional computational costs compared to the current state-of-the-art algorithm.