Tackling Environment Heterogeneity in Federated Reinforcement Learning

We investigate a federated reinforcement learning (FRL) framework, particularly in contexts where local environments exhibit heterogeneity. Within this framework, we propose a novel method designed to ensure stable performance across all local environments, along with their plausible variants. The central concept involves the development of a robust local update mechanism, effectively addressing potential risks arising from the heterogeneous local environments of others and from unexpected perturbations. Furthermore, we introduce a pessimistic Q-function, which facilitates the extension of our approach into large or continuous state spaces. Through experiments, we substantiate the effectiveness and robustness of our method in heterogeneous environments, thereby verifying its adaptability and reliability across diverse applications.