Enabling Adaptive Pilot Assistance Through Task-Awareness: Evaluation of an Activity Recognition System

This contribution evaluates an activity recognition system developed for the multitasking-heavy work domain of pilots. Due to the numerous supervisory control tasks in modern aircraft cockpits, pilot performance is severely affected by sequential multitasking and context switching, which makes automated activity recognition a severe challenge. Therefore, we proposed an activity recognition method based on the sequential analysis of the pilot's cockpit interactions. Hence, we developed an agent that encodes the pilot's cockpit interactions into a series of feature vectors. Then, the pilot's tasks are classified in real time using probabilistic graphical models (PGMs). Since this paper focuses on evaluating the methodological approach, the activity classifier was not trained but parameterised using an automated approach based on expert knowledge. We conducted an evaluation experiment with five military fighter pilots in a fast jet simulator. The results indicate that smaller window sizes have a higher variation in classification performance. Additionally, the performance of the classifier varies greatly between pilots. The best results were obtained with a single-layer PGM. In general, the classifier has moderate to good performance, suggesting that the classifier would benefit greatly from training. The proposed activity recognition system will be integrated into a task-aware adaptive assistance agent. The agent is designed to assist fighter pilots who manage unmanned aerial vehicles directly from the cockpit.