Robust Human-Machine Teaming Through Reinforcement Learning From Failure via Sparse Reward Densification

Learning control policies in sparse reward environments is a challenging task for many robotic control tasks. The existing studies focus on designing reinforcement learning algorithms that take human inputs in the form of demonstrations such that control policies are learned via uncovering the value of these demonstrations. One typical approach is to learn an inherent reward function that can explain why demonstrations are better than other randomly generated samples. Albeit powerful, the use of human demonstrations is typically costly and difficult to collect, indicating the lack of robustness in these studies. To enhance robustness, we here propose to use failed experiences, namely, failure, due to the easiness of obtaining failure dataset, requiring only common sense rather than domain knowledge needed to generate expert demonstrations. In particular, this paper proposes a new reward densification technique that trains a discriminator to evaluate the similarity between the agent’s current behavior and failure dataset provided by humans. This reward densification technique provides an effective mechanism to obtain state-action values for environments with sparse rewards, via quantifying their (dis)similarity with failure. Additionally, the value of the current behavior, formulated as advantage function, is employed based on the densified reward to refine the control policy’s search direction. We finally conduct several experiments to demonstrate the effectiveness of the proposed approach by comparing with state-of-art methods.