"Guide Me Through the Unexpected": Investigating How Deviation from Expectation Affects Human Teaching and Robot Learning

The increasing integration of robots into human environments necessitates efficient learning systems capable of adapting to complex scenarios while co-existing with humans. Traditional reinforcement learning (RL) is one of the most popular option, but often struggles with inefficiencies, such as sparse rewards and prolonged training. Learning from Demonstration (LfD), which leverages human expertise, offers a promising alternative. However, human teaching strategies and robot learning processes are inherently intertwined in LfD. Ineffective human teaching can diminish robot learning. To effectively provide demonstrations, human teachers require an understanding of the robot’s internal processes and needs without being overwhelmed. We address this by visually showing the robot’s deviation from expectation, a metric based on Temporal Difference (TD) error, which represents discrepancies between predicted and actual outcomes. We conducted a user study (n=12) comparing two conditions: one in which deviations from expectation were visually indicated, and one in which these deviations were not shown. Results indicate that visualising deviations shifts human teaching behavior from result oriented strategy (providing demonstrations in the areas where the robot fails) to an expectation oriented strategy (focusing on demonstrations where robot’s deviation from expectation is high). We conducted a follow-up simulation study to investigate how these two teaching strategies may influence robot learning, showing that diverse and widespread demonstrations have a significant effect on robot learning performance. We conclude our work with actionable guidelines for designing human-robot interactions that better align human teaching behaviors with robot learning requirements.