Language and Gesture in Virtual Reality: Is a Gesture Worth 1000 Words?

Robots are increasingly incorporating multimodalinformation and human signals to resolve ambiguity inembodied human-robot interaction. Harnessing signals suchas gestures may expedite robot exploration in large,outdoor urban environments for supporting disaster recoveryoperations, where speech may be unclear due to noise or thechallenges of a dynamic and dangerous environment. Despitethis potential, capturing human gesture and properlygrounding it to crowded, outdoor environments remains achallenge. In this work, we propose a method to model humangesture and ground it to spoken language instructions givento a robot for execution in large spaces. We implement ourmethod in virtual reality to develop a workflow for fasterfuture data collection. We present a series of proposedexperiments that compare a language-only baseline to ourproposed language supplemented by gesture approach, anddiscuss how our approach has the potential to reinforce thehuman’s intent and detect discrepancies in gesture andspoken instructions in these large and crowded environments.