Adaptive Weighting by Sinkhorn Distance for Sharing Experiences between Multi-Task Reinforcement Learning in Sparse-reward Environments

Abstract: In multi-task reinforcement learning, an agent using off-policy learning can leverage samples from other tasks to improve its learning process. When the reward signal from the environment issparse, the agent in each task spends most of its training time exploring the environment. Therefore,the shared experiences between tasks can generally be considered as samples derived from an exploration policy. However, when the exploitation phase begins, the shared experience framework mustaccount for the divergence of policies across different learning tasks. However, when the exploitation phase starts, the sharing experience framework has to take into account the policies’ divergenceissue of different learning tasks. Our work addresses this issue by employing an adaptive weight forshared experiences. First, a central buffer collects and shares the experiences from each individualtask. To mitigate the effects of policy divergence among multiple tasks, we propose an algorithm thatmeasures policy distances using the Sinkhorn distance. The computed distances are used to assigna specific weight to each shared sample, controlling the amount of knowledge shared as the policiesbegin to diverge during the exploitation phase. We conduct experiments in two goal-based multitask learning environments to evaluate the effectiveness of our approach. The results show that ourproposed method can improve from 8%-10% in average rewards in comparison with other baselines.Keywords: multi-task reinforcement learning, off-policy, experience sharing.