HRI-SENSE: A Multimodal Dataset on Social and Emotional Responses to Robot Behaviour

We introduce HRI-SENSE, a multimodal dataset of Human-Robot Interactions (HRI) studying users' social, phys-ical (e.g. facial expressions, body movements) and emotional, psychological (e.g. frustration, satisfaction) responses to robot behaviour. The dataset captures participants collaborating with a TIAGo humanoid robot following various behaviour models on a manipulation-based “Burger Assembly“ task, eliciting different user reactions. HRI -SENSE contains over 6 hours of verbal and physical interactions taking place over 146 sessions with 18 participants, recording multiple modalities captured simultaneously by RGB and Depth cameras from three angles and one microphone. The time-synchronized multimodal data include non-verbal behaviours (e.g. facial landmarks, expressions, pose landmarks), explicit feedback signals (e.g. verbal interactions), robot movements and self-assessed questionnaires on sociodemo-graphics and user impressions (e.g. frustration, satisfaction) on robot interactions. HRI-SENSE is expected to facilitate further research into modelling non-verbal behaviour and advancing the development of user-aware interaction models in HRI domain.