A Multimodal Virtual Reality Data Acquisition Platform and Dataset to Assess Systemic Human Cognitive States

In recent years, there has been an increasing interest in human-machine teaming for search and rescue operations, deep space missions, and agricultural tasks, among others. To be effective teammates, artificial agents should be able to detect and be responsive to systemic human cognitive states such as workload, sense of urgency, mind wandering, interference, and distraction. Here, we introduce an experimental paradigm and a comprehensive multimodal dataset that provides the necessary data for analyzing the relationships among multiple systematic human cognitive states, enables the development of robust prediction models of these states, and details the framework for developing new experiments. The introduced experimental setup allows for the synchronized real-time recording of multiple data streams from various sensing devices including fNIRS, EEG, pupillometry, respiration, electrodermal activity, and plethysmography, and is applicable in many other interactive task settings where human performance needs to be monitored. The dataset was acquired from 80 subjects performing a driving task and several secondary tasks including car braking events, dialogue communications, and tactile stimulations.