Predicting Human Context-Aware Action Modifications for AI Assistance in Visually Demanding Tasks

Abstract For a plethora of humanitarian and commercial applications such as medical imaging, air traffic control, driving, and supervisory control, artificial intelligence is assisting humans in increasing performance in visually demanding tasks. To design an effective AI team, AI must provide the proper level of assistance predicated on human cognition and performance. To achieve this, it is essential to monitor and predict human performance in real-time rather than relying solely on overall task performance. Reaction and decision time can be utilized to predict the mental workload of humans in visually demanding tasks. In this paper, we utilize the benchmark Atari environment to remove domain expertise and provide emphasis on context-aware human decision-making. We extract connected component labeling features from the environment and human eye gaze to predict when a human performs a context-aware action modification for both frame-by-frame and time-domain applications. We identify specific environmental scenarios where eye gaze provides a wealth of information and increases real-time workload classification. The classification results are analyzed for specificity, sensitivity, and F-score to illustrate the mitigation of misclassified information. The results demonstrate the effectiveness of utilizing environment features in combination with eye gaze to predict when the human needs AI assistance during a visually demanding task.