Development of multimodal augmented reality applications on the flight deck for single-pilot operations: subjective user satisfaction

Abstract In the current commercial two-pilot flight deck, the pilot-flying (PF) is responsible for flying the aircraft, and the pilot-monitoring (PM) focuses on communicating and monitoring PF’s operational behaviours. The driving factors behind single-pilot operations (SPO) are the foreseen pilot shortage and the desire to reduce operating costs. Novel technologies incorporating voice and gesture interactions within an augmented reality (AR) interface has garnered interest as potential enabling technologies of SPO. The current study compared the usability and user satisfaction of voice- and gesture-command functionality for interacting with AR information which guided the sequential completion landing tasks. Forty participants completed two automated landing trials whilst interacting with AR-presented task checklist prompts, completing five procedures of the landing checklist using both voice and gesture commands. Results revealed that interacting with AR information with voice commands, when compared to gesture commands, led to higher user satisfaction (measured via the Questionnaire for User Interface Satisfaction), which indicates a perceived higher subjective usability (measured via the System Usability Scale). Participants reported voice-commands to enhance flying performance compared to gesture-commands, as the latter created unnecessary burden upon cognitive resources and interfered with the physical execution of the main manual flying task. The application of virtual holographic checklist with guidance cues can improve pilots monitoring performance and procedure compliance during instrument landing trials of this study. Future research should explore the implementation of combining AR presented information with voice-command interactions on the flight deck to support a single pilot performing both flying and monitoring tasks.