Deep Learning Forecast of Attention Using fNIRS Data

Attention is a critical cognitive resource that supports both working memory and situation awareness (according to human information processing theory). Some aviation incident reports have also referred to these cognitive constructs in describing/explaining piloting errors. Thus, advanced in-cockpit monitoring approaches have been introduced to monitor pilot attention during flight tasks. However, most of these systems are reactive in nature, aiming to predict current attention levels. This work introduces a novel methodology for forecasting future attention load shifts using pilot fNIRS data. To this end, a small sample of seven participants with varying degrees of flying experience were recruited and tasked with flying a UH60 V Blackhawk helicopter simulator in which various task conditions were introduced to simulate real flying scenarios and manipulate levels of attention. An established multinomial classification model was subsequently applied to classify pilot attention states using fNIRS features obtained from the experiment trials. Following this, a baseline statistical model and six deep learning models were developed to forecast changes in the predicted attention level. An ARIMA model, two linear models, and four recurrent models were leveraged. All deep learning models outperformed the baseline statistical model. Recurrent models that included a differencing processing element were most effective at predicting future shifts in pilot attention. Incorporating cross-attention with fNIRS features did not improve the recurrent model performance. Future forecasting work should consider the first order differencing of classification confidences in their modeling for improved performance.