Evaluating User Behavior Consistency: Comparing Studies in Virtual, Crowdsourced and Real-World Human–Robot Cooperation Scenarios

Human–robot collaboration enables highly adaptive coworking. The variety of resulting workflows makes it difficult to measure metrics such as, e.g. makespans or idle times for multiple systems and tasks in a comparable manner. This issue can be addressed with virtual commissioning, where an arbitrary number of nondeterministic human–robot workflows in assembly tasks can be simulated. To do this, data-driven models of human decisions are needed. Gathering the required large corpus of data with on-site user studies is quite time-consuming. In comparison, simulation-based studies would allow us to access a large pool of study participants with less effort. To rely on respective study results, human action sequences observed in a browser-based simulation environment must be shown to match those gathered in a laboratory setting. To this end, this work aims to understand whether cooperative assembly work in a simulated environment differs from that in an on-site laboratory setting. We show how a simulation environment can be aligned with a laboratory setting in which a robot and a human perform pick-and-place tasks together. A user study (N = 48) indicates that participants’ assembly decisions and perception of the situation are consistent across different environments, including a scalable crowdsourcing setup.