Robot movement based inference of teleoperator state for multi-robot search

Human behavior encodes important information about how we perceive and act in a new environment. Extracting such information indirectly from behavioral observations has the potential to advance natural human–robot interaction without the need for wireless transmission of sensitive data. This is especially relevant in wilderness search and rescue missions that use unmanned aerial vehicles (UAVs), to cover large areas. In such missions, while human teleoperators typically have prior knowledge about the environment or the missing person, they also experience mental workload, and may not be situationally aware at all times. An effective human–robot team, where teleoperators are assisted by autonomous robots, should function so that the autonomous robots are able to adapt their search strategy in response to human state—knowledge, workload, and situational awareness—based on movement of the teleoperated UAV. Accordingly, in this paper we test the hypotheses that teleoperator knowledge and state can be estimated from movement cues and that such cues can then be used to adaptively steer autonomous robots toward an improved search performance. We conduct human-subject experiments in a virtual replica of a section of the Grand Canyon National Park, which reports one of the largest number of missing person cases every year. Participants were selectively briefed with prior knowledge about the local terrain and the missing person before every trial. EEG and eye tracking data were collected to directly measure cognitive load and situational awareness through multiple measures. Our results show that both search performance and turn rate of teleoperated vehicle depended on missing person knowledge. We further find that cognitive load and situational awareness as measured through physiological measures are weakly correlated with movement behavior. To use these results for adaptive search, we estimate prior knowledge and situational awareness from movement data and use it in a human-swarm robot interaction strategy where multiple robots adapt their separation and proximity to the human teleoperated UAV. Our simulation results show that adaptive search performs nearly as well as a comprehensive spiral based search in terms of success rate and is able to do so in less time. Results from this work set the stage for new non-verbal human robot interaction strategies in field settings.