Interaction-aware augmented reality interface for assisting erring social robots

Abstract In the future, many service jobs will be performed by socially interactive robots. When initially introduced or still learning, such robots may sometimes make incorrect decisions or perform erroneous actions, thereby causing socially inappropriate interactions with guests or customers. To prevent potential problems during interactions, a human worker could be present to help or intervene when necessary. Such assistance would occur directly in front of customers, so the interface used by the human clerk is crucial. In this work, we study the use of an augmented reality (AR) head-mounted display for human clerks to oversee the robot. With such an interface, communication between the human clerk and the robot can be largely hidden from customers, especially in comparison to a traditional screen-based interface. Nonetheless, the interface may still lead to unnatural head movements by the human clerk, potentially negatively affecting customers’ impressions of the interaction. We address this by proposing an interaction-aware gaze guidance method for the AR interface, where the AR objects are moved to ensure that the resulting head movements appear natural. A user study in a sales scenario demonstrated that this method improves both customers’ subjective experience and the usability of the interface.