Adversarial Inverse Reinforcement Learning with Noisy Observations

Inverse reinforcement learning (IRL) has emerged as a popular approach for training robots from human/expert demonstration, where a learner/robot infers the expert's hidden reward function using the demonstrations and a simulator. We argue that noise is inevitable in certain parts of the demonstration, and show that such noise does indeed deteriorate the performance of a popular and widely applied IRL method, called Adversarial IRL (AIRL). To render AIRL robust to noise, we formulate the problem of reward inference as one of log-likelihood optimization that accommodates noisy input. We adopt two techniques from the literature on learning hidden representations in sequential decision tasks and combine them with AIRL to solve this unified optimization problem. Experiments in four benchmark OpenAI Gym environments show that our proposed methods are effective in overcoming demonstration noise for the task of reward learning, but less so for the task of reproducing the expert behavior.